Both my wife and I have the iPhone 4. The upgrade to iOS 5 has been a terrible experience. The upgrade itself took an inordinate amount of time. Sync process afterward had to be run multiple times before everything showed up and ran. First attempt had every app (except built-in apps) simply flash and go away, but still show running. It required another sync to fix that.
The biggest issue at the moment is I can't trust email anymore. After upgrade, my inbox was empty except for the email Apple sends about the cloud setup. Fortunately I had the emails on my laptop. Sending or replying to a few emails now has them not showing up in the sent items folder. So I don't really have confidence they went. Testing emails between wife and I on our phones has occasionally had them not go or take a good 10 minutes to get from one to the other. Huge problem for relying on it for work email. Wife has a bunch of emails with no sender that say they're from 1969. Nothing in them and she can't get rid of them.
And now when I plug the phone in and it starts to sync, it still mentions transferring music files. I haven't changed any music on the phone or laptop, yet it still wants to copy more.
Videos, books and ring tones my wife had paid for and downloaded still don't want to transfer to the phone after multiple syncs and reboots. So syncing appears to be jacked up.
All in all, this upgrade has been a terrible experience.