For the H264 videos, there isn't a defined maximum bitrate. I have 2 movies encoded at 18mbps and 14mbps respectively and they play fine (most are 4mb-6mb). It'll just eventually fail to keep up. It reads 1080p videos fine...it just plays them slowly.
Surround music (DTS-A) has to be downcoverted to 48khz/24bit 640kbps AC3 to output through it.
The ATV2 will output lossless/uncompressed music files at that same resolution. My receiver indicates that the 96/24 and 192/24 and 352.8/24 Nordic 2L files get sent to the receiver at 48/24bit.
To appreciate 96/24, you need some serious audio gear and serious fanaticism though (have a friend play 96khz then 48khz...if you can't pick em out, you're wasting money). The kind of device that sends faithful 192khz+ signals to a receiver that knows what to do when them and speakers that can do it has an entry price of insanity insanity also being the salient personality trait of an individual who can pick 192khz out of a lineup of 96khz music.

Most of my friends can't even pick out a 2.8224mhz SACD.
If you're that lunatic, I feel your pain. SACDs shouldn't have died out. They're amazing. The digital-download revolution and Best-Buy-surround-sound-in-a-box killed high fidelity. Good luck. Whatever garbage audio device they use in the ATV2, it couldn't process 96khz anyway. The PS3 does a moderate job of it, but completely fails to distinguish 192khz with any grace.
(unless you have nearly 6 figures worth of speakers and amps, you can carefully downconvert 192khz to 48khz that are virtually indistinguishable...do it in 32bit resolution in Sountrack Pro or something...I got a few converted that just lost a tiny bit of staging and warmth on Krell gear driving Sonus Fabers)