Personally, I don't see a downside to sharing all of my media and having access to all of it. I would hate to have to go to my mac manually and change what is being shared and what isn't. So what are your concerns for just sharing your entire iTunes library? Is it taking a really long time to load in?
I have about 2600 songs, and 1000 photos, and only a couple of TV shows. So it is pretty much instantaneous. It still spins on screen saying that it is loading my library, maybe for about 5-7 seconds, but it is not bad at all. I suppose if you have a lot of videos then maybe it takes longer?