I think I've finally cracked it. That is converting HD-DVD to a format compatible with both AppleTV and the PS3. I just converted a 20 second, 720p file and it is playable in Quicktime.
Here is the method I have used. Every step is completely necessary if you want your file to play perfectly. You will need a lot of Hard Disk space because the video needs to be changed into different containers 3 or 4 times.
1. Once you have ripped the Disc using AnyDVD HD you will need to use EVODemux to extract the VC1 stream and the .Eac3 audio stream.
2. Convert the audio stream to .ac3 using Eac3to. Instructions for this are included in the guide I posted a few posts back.
3. Convert the .vc1 stream to 23.976 fps using vc1conv. This is a command line tool but is not very complicated. Can be found
here
4. Then you need to go through a few steps which allow the new stream to be read by popular video encoders (I'm using Super). The process is VC12avi ---> VirtualDubMod ---> SolveigMM ASF Muxer. This
guide goes through it in more detail. Keep in mind that each of these steps will create a new file of about 15-20 GB. Each step will take 30 minutes approximately.
5. After ASF Muxer has done its job you should be left with a .asf file. This file can be read by
Super.
6. Then just use the GUI to set the size 1280*720, framerate 23.976 and the bitrate of your choice. For AppleTV you will also need to change the h.264 profile level to 3.1 by selecting the h.264 profile button. Disable audio because we haven't got any at this stage). Also, right-click in the drop box and select "output file save management" to save your converted file in the right spot. Conversion to 720p will probably take 5-6 hours
7. Once the conversion of the video is finished you will need to mux in the audio (.ac3) from earlier. Now I'm not sure of the best way to do this but it is possible to just add the .ac3 by copying and pasting it into the video using Quicktime Pro (Add to Selection and Scale). If you save the new file you should have a 720p .mov stream with 5.1 audio. Visualhub may be necessary for AppleTV output.
I estimate the total encoding time will be roughly 8 hours from demuxing the original video and audio streams to having your file playing on your AppleTV.