My home machine has 2008 (no problems so far with SL), but two at work (one wifey's) have 2004. A reinstall should work fine but you need to remove bits and pieces, not just the 2004 MS Office folder. You need to remove the prefs and the Microsoft folder in the user folder>library>preferences. I would leave the identity folder so that your Entourage prefs would be preserved.
Since I am also not happy with the outcome (had to reinstall twice), I am looking at a bargain deal for Office 2008. The best deal for Special media edition (it has Exchange support) is on Amazon marketplace - under $100.


Alex
3.1 GHz 13" MacBook Pro 2015, 8 GB RAM, OS 10.11.2, Office 2011, TimeWarner Cable
2.8 GHz Xeon Mac Pro 2010, 16 GB RAM, OS 10.11.2, Office 2011, LAN