The 20" model has the effect of a lower gamma (I think that's right, lower=brighter) towards the bottom of the screen - this means that particularly grey colours can look a bit lighter towards the bottom than they do at the top.
An easy way to verify this is to open a Safari window and check the shade of grey at the top, then drag the window to the bottom of the screen and see if the shade is lighter there.
This is actually not a fault but a characteristic of the "TN" type display panel used. Indeed I'd never noticed it until the iMac (because it's such a big and glossy screen) but I went and looked at my Powerbook from 2004 and it's exactly the same. The screen I use on my PC also "suffers" from it to an extent. I strongly suspect all 20" Al-iMacs are like this, and people who say theirs isn't are either in denial (normal for an Apple forum) or very lucky.
Now that I've seen it's not a fault I'm not bothered about it.
The 24" uses a different technology, which doesn't have the top-down fading. Instead, people complain about a slight discoloration from left to right.
Now, I was in PC World (who ironically sell Macs) just a couple of days ago and they had a 20" and 24" side by side. Unsurprisingly, the 20" displayed exactly the same top-down fading. Bear in mind you have to be looking for it to notice it most of the time, and it only really affects greys etc.
The 24" was perfect top-down but if I looked really carefully I could see that the grey colour of the Safari titlebar was a *tiny* bit more purple on the left and right and a tiny bit more yellow in the middle. The difference is nowhere near as bad as the 20" top-down, and I could only barely notice it, and was trying particularly hard. TBH, if people are complaining about it and theirs are only to the same extent as this, as in it's 0.00000001% more yellow in the middle, they are *incredibly* fussy. Again I suspect it's probably a fact of the technology rather than a design fault.