I think it is because the remote can be used with your mac too. I was wondering why stuff kept playing on my mac when I first got my ATV as well.
If you open System Preferences, go to Security & Privacy, then click the "Advanced..." button at the bottom, you can disable the remote control on your mac. I'm using Mountain Lion, so not sure if it will be in the same place on older versions.