One of at the things Handbrake does is convert your 1st audio track to AAC stereo. The aTV looks for that track. I would suggest taking a look at the files you have that work and see if the 1st audio track is the same or different than 1st track your problem file.The ATV is connected directly to my Home Cinema via optical out and performs perfectly for all other files I have converted using Handbrake. I can change settings and use a pseudo surround, but I would prefer the proper separate channels. This problem file converts and works perfectly with 6 channels when converted from MKV to m4v using Handbrake, but not when using Subler. The only other MKV file I have converted with Subler worked fine, so there must be something odd about this problem one.
I will see what happens on the next MKV I try to convert.... I'm not giving up on Subler yet ;-)
----------
guys, keep in mind just because something is h.264 doesn't mean it's compatible. Apple tv (and probably other things like itunes) only support playback of certain h.264 profiles/levels. Chances are the h.264 video in your matroska container isn't a supported type (it rarely ever is in my experience).
This is why converting it via handbrake profiles always works; with the right profile, it will always produce a video stream that your device is capable of decoding.
If the h.264 stream in the container is not one of the supported profiles/levels, i'm afraid there is nothing else to do but convert it to one.
+1