I went through the buyer's remorse too, partly due to comments on boards like these. The day I bought at the Apple Store, I looked at both models and could see that the 24' display looked a little better, but I decided that it was not $300 better, and I didn't need or really desire the extra screen space.
I wasn't crazy about the 20' display once I got home, but color calibration resolved my issues. I use the 2.2 gamma, native white point, and I have the brightness dialed down to about 40% of full. It works for me. Yes, there are distortions when looking at an angle, but it's not how I look at my screen.
During the first 5 days or so of my discontent, I did seriously think about returning the iMac, but I didn't because of the restocking fee, hassle of bringing it in, etc. Tomorrow will be day 14 of my purchase, and I'm happy with it. No intent to return it.
If I had to do it all over again, I might go with the 24' screen. It would be a toss-up again. But despite the naysayers, I have grown to like my 20' model more and more, and I'm very satisfied.