Anyways, the biggest improvement for me in the new Office suite has got to be Outlook 2003. The new vertical-pane UI makes it look much more un-cluttered, and Bayesian filtering is a God-send
As you said, the individual components like Word, Excel, Powerpoint etc. load very quickly, and I'm sure XP's prefetching contributes to that as well.
The new icons (in the Start menu as well as in the apps) are a nice, bright change from the old Office XP icons, but nothing phenomenal nonetheless.
The whole XML document format (like OpenOffice) is cool too, but the addition of some other useful features from OpenOffice's Writer (such as Export to PDF without Acrobat installed, auto-completion of words by just hitting enter) would have been great.
So far I've only really used Outlook and Word on a regular basis, so I don't know much about what's changed in Excel and Powerpoint. As for Frontpage, I'll probably never use it since I'm not a fan of WYSIWYG editing.
But yeah, I think it's a very nice upgrade. The changes in Outlook alone make it worth it.