It really depends on what you're doing with them. I have a 1.66 Mini, and for the most part, yes, its very quiet. But, at times it can get a little loud. Its mostly related to graphically intensive stuff. I'll show you what I mean, with some help from iStat Menus...
Right now, just browsing the web and stuff, its almost dead silent. The fans are barely moving.
But, if I start watching an HD video of any kind, be it a trailer or a podcast, the fan will kick up a bit.
But, what really gets it going is anything that uses the OS effects heavily. The one that I see all the time is when flipping through iTunes in Cover Flow view. Here is what happens after just one pass through my albums, at a moderate pace, like I would if I was looking for something to listen to...
Now, the thing to keep in mind is, if you have anything AT ALL going on in the room where the Mini is, you will hardly notice this sound. Even at almost 5,000rpm. I'm watching SNL right now, at a slightly low volume, and I can't even hear the Mini. But, if its quiet, you will hear it. Its not bad, and it goes back down quickly.
I think its more of a preventative thing than anything. Its like the OS knows that these effects are hard on the CPU/GPU, and kicks the fan up quickly to keep the heat down. Then, after the effects stop, the fan can quickly go back down. After flipping through iTunes like that, my temps actually go DOWN, because the fan goes so fast.
But overall, like I said, its hardly noticeable. Unless you only use your machine in a dead silent environment, you might never even notice the noise. Media streaming shouldn't be a problem. Playback of HD material might a bit. But, if you are just going to use it to send media around the network, no problem.