That was going to be my answer as well. On the computer whose library you don't want to see, remove AppleTV from that computer in iTunes (Preferences / AppleTV).
However, if someone comes over with their laptop and I link them with AppleTV, they'll stay on the AppleTV list and I don't know how to remove them. I just tested this - I linked an old PC laptop with my AppleTV, and it's showing up under sources even though I'll never really use it. If the PC laptop died or if I gave it away, I don't think I could remove it from the AppleTV Sources list (without a factory restore).