i have spent nearly two years in this nightmare --how to create a single file that plays in 5.1 on both the atv and ps3.
recently i finally hit pay-dirt after trying every single trick mentioned and then some. i have changed the atv preset h.264 options to include level 41, changed all the file names to .mp4 to suit the ps3, made sure the file was created in the correct mpeg container that ps3 likes (whatever that means), used mpegstreamclip/ ffmpegx/ dvision3/ visualhub/ toast/ roadmovie/ and others to check HB wasn't the issue, ripped with DVD2oneX instead of MTR, ripped with RipIt, and on and on and on and on and on and on and on ... ... ... some worked, most didn't. nothing was ever 100%.
and then just before i hacked the atv and was done with it, someone in a forum gave me the answer. it was really quite simple: using the HB 9.3 PS3 Setting, click the Audio/ Subtitles Tab, change Track Mixdown to 6-Channel Discrete. voila. bliss.
and now HB have updated and the ps3 setting is gone.
ps3mediaserver transcoding on the fly is a great solution. but not 100%.
currently i am using the appleTV setting with a second audio track added -ac3 with 6-channel discrete. i do get 5.1 sound from both the atv and ps3 but i suspect my yamaha amp has the most say. again, not 100%, but in the 90's.
sony should just fix the problem --the lastest firmware upgrade included Polish as a language. and something to do with albums for photos.
the new Samsung series 8 TV's come with an in-built media player that plays every file type ever. plug is a hard-drive and that's it, push play. soon, no-one will need atv/ ps3. particularly with all the grief they give us.