OK, here's what worked for me on my Mac using just Visualhub, mkvextract (for ac3), VLC (for dts), and Quicktime Pro.
First, I should clarify what I want to do. I want to play 720p mkv's on my hacked Apple TV while preserving the multi-channel sound and not just transcoding to stereo. There's no avoiding converting the video to a format that the ATV can handle... the bitrate on the original is too high.
For ac3 audio, I did everything Stomer suggested except I was able to skip the step with MP4Box because I could open the mkvextract'ed audio file directly with QT. You could even avoid using mkvextract to extract the ac3 audio by using QTPro but it's slower because you have to wait for the mkv file to fully load before you can extract the audio. I saved the movie as a Quicktime Movie rather than export because I wanted to preserve the multi-channel audio.
The ATV cannot decode ac3 or dts. It just passes them through the toslink port so a dts/ac3 receiver can decode it. My ATV is currently hooked up directly to a 720p lcd in my bedroom but I can hear the audio on ac3 movies because I have an ac3 decoder installed using NitoTV (a plugin for ATV). dts is more complicated. If you're going to hook up your ATV to a receiver that can decode ac3/dts, then for a mkv with dts, just open the mkv file in QTPro, wait for the file to finish loading, extract the audio and paste it into the video file you converted with Visualhub. If you don't have a dts decoder, then you have to use VLC on the mkv file, and extract the audio as an mpeg-4 file using the wizard. Open the audio mp4 file in QTPro, open the movie properties and delete the video track (for some reason, VLC includes a small video even if you choose to transcode audio only). Select all, copy, and paste the audio into your video. You can either export to mp4 for just stereo audio or save as a Quicktime movie giving you 6 channel aac that the ATV (I assume) can downconvert to stereo. However, AFAIK, most audio receivers can't decode AAC (at least mine can't but it's 4-5 years old).
You may be asking yourself why I'm preserving the multi-channel audio when I've only got my ATV hooked up to an lcd with just stereo sound. It's because I'll either eventually move my ATV into my living room where my dts/ac3 receiver is or buy another ATV for my living room.