Exactly. I ordered a silicone case for mine (Incipio Dermashot), but it's black, so dust and/or minor scratches won't be seen. Just being able to see a spec of dust would piss me off.
I recognize that if dust can get under an iPhone care, more dust will be able to get under a larger iPad case.
I elected for a silicone case purely because I think it'll be more comfortable to hold for long periods than a plastic case would be (like my Sedio Innocase on my iphone). I'm willing to trade some minor scratches in exchange for comfort.
Actually I've noticed that the darker colors show dust more than the lighter colors, mainly because dust itself is light colored and stands out more on a dark background.