Something I found suggested on the Apple forums was to make sure the machine hosting your library and the ATVs are all on the same sort of connection - i.e. all wireless or all wired.
my setup is all wireless to the airport extreme.. still have the problem. I told the iMac not to sleep and it *may have helped*, but not fixed it. All you people make sure you log the problem with Apple so maybe they could carve out a few hours to find the bug. Maybe when the iPhone 5 is out they can spare an engineer.