The problem here is that your movies with 5.1, they are encoded in Dolby Digital 5.1 aka AC3. The iPod Hi-Fi is expecting a different type of digital signal. The Hi-Fi is expecting a PCM digital signal (S/PDIF standard) over the optical connection (Toslink.)
Basically, the optical connection on the appleTV will send the PCM digital signal when listening to music and maybe other AppleTV stuff (podcasts, youtube, etc)
BUT... When you play that movie file with 5.1 sound, the appleTV is sending a Dobly Digital signal (AC3) over the optical connection, one that the iPod Hi-Fi cannot decode.
You may? be able to switch off the 5.1 signal while retaining it in the movie file. I think? you can do this with Quicktime Pro. If this works the the AppleTV will? play the stereo aac sound out of the optical port as PCM digital signal that the iPod Hi-Fi can decode properly.
The confusing thing for most people is that many different signals can be sent over Toslink optical digital connections. PCM/Dolby Digital/ADAT/DTS just to name four.
The iPod Hi-Fi can only accept PCM digital(S/PDIF aka Sony/Philips Digital Interface) or analog
The AppleTV can send PCM or Dolby Digital and....
I could be totally wrong here, but I think when you put a Dolby Digital (5.1 or stereo) audio track in a movie file, then the AppleTV sends that signal thru the optical out and sends the aac audio track (if included in the 5.1+2 Visualhub setting) out of the RCA connectors.
Hope this helps, and if I got any of it wrong, I hope someone else can correct me.
Additionally:
From my understanding? (could be wrong), maybe Dynaflash can add some here:
AppleTV can only play 5.1 sound from .mov and .m4v files
If your file is .mp4 then the AppleTV will send out your stereo aac track out of the optical output in a PCM signal which your iPod Hi-Fi can easily playback.
You should try changing the extension of your .mov files to .mp4 then maybe this will play back all your movies in stereo on your Hi-Fi. Your files will retain the 5.1 DD and you can change their extensions back to .mov or .m4v when you upgrade your sound system.
Good Luck