I have had the exact same issue with both Subler as well as MKVtools.
I haven't researched extensively, so please forgive any ignorance, but I think the problem lies in the H.264 profile. The codec is compatible, as we know, but most MKV video tracks use a profile that is not playable on Apple TV. It doesn't seem like there's any way to change the profile without doing a full re-encode (a la Handbrake). If someone is well-versed in the codec/profile world, please share!
I did a final test last night with a 1.5GB MKV file (a 30 min show). I created two M4V files, one with MKVtools and one with Handbrake. It took 5 minutes for the MKVtools file and 1.5 hours for Handbrake (MacBook Pro, 2.4 GHz Core 2 Duo, 4GB RAM). As expected, the MKVtools file appeared on the Apple TV, but displayed the error message when I tried to play it. The Handbrake file played flawlessly.
At this point, there are a few questions:
- Is it the H.264 profile that's causing the problem?
- Is there a way to update the H.264 profile without re-encoding?
- Is it an issue with the audio track (I believe MKVtools re-encodes the audio from AC3 to AAC, but can't confirm)?
- Is there a simple setting in MKVtools that I'm unaware of that solves the problem?
- If all hope is lost, are there optimal settings for Handbrake that speed up the re-encode process?
Just to make sure we're all on the same page, a good way to think of a video's avc profile is as a specification for what properties are allowed for that movie. A High 5.1 video can be encoded using more fancy settings than a Main 3.1 video, because the Main 3.1 profile is more restrictive. A few things to point out here. (a) Once encoded, you can't change the properties that make a video High 5.1 without re-encoding, (b) a High 5.1 video doesn't have to be encoded using the fancier settings, and (c) all it would take is for one of the "fancier settings" of a higher profile to be an problem, for a manufacturer to have to specify a lower profile is needed for video playback.
I'm not sure about Subler, but here is what is going on with MKVtools. When it passes through video, it looks at the target device, and then edits the movie at the hex level to adjust the avc profile "label" to an appropriate value. So when the target device tries to play the video, it's fooled into thinking that the profile level is appropriate.
A PS3 is happy with High 4.1. Most MKV files are High 5.1. There is not much difference between these two profiles that would show up in common encodes. So when the PS3 is fooled into playing a High 5.1 video that has been relabeled to High 4.1, then there usually is not issue. Unless I need to burn subtitles, I've never had to re-encode the video track for playback on PS3.
The Apple TV, however is more restrictive. It wants a Main 3.1 profile. I don't have an Apple TV yet but I do have an iPad which has the same video requirements. The movies I tested with the newest version of MKVtools, using passthru for the video, played fine, but then I don't have hundreds of videos laying around to test (that's where beta testing is a big help

). As I see it, there are two things that could be going on if the ATV won't play videos with a passed thru video track:
- MKVtools did it's job, and the Apple TV is fooled into trying to play the video, but there was some setting when the video was encoded which is specific to a profile higher than Main 3.1, and ATV can't handle it. If this is the case, then there is nothing that can be done other than re-encoding the video.
- MKVtools did not fool Apple TV into trying to play the video. If that's the case then there may be a fix.
By the way, this discussion seems a bit off topic for this thread, so it might be more appropriate to start a new thread, or e-mail me if anyone wants to pursue this goal of being able to passthru mkv video tracks for playback on the ATV. I don't have access to these problem videos, so if you do and you want to find a fix, here are some things that you can try. By the way, the latest version of MKVtools is 2.4, so make sure you are using that.
- Use an app like mediaInfo to check the avc profile of the created m4v video. If it hasn't changed, then that would be the issue.
- It was mentioned that an older version of MKVtools did successfuly passthru the video tracks of these problem files. Can that be verified?
- To rule out the audio as an issue, go to the mp4 tab and check the option that will re-encode the video for ATV. Make sure your resolution is not greater than 1280.