It will likely take a revision to Handbrake. There are only two containers that can be generated with AC3 Dolby Digital 5.1 with Handbrake - .mkv and .avi. Neither of these play on the current ATV, nor is it likely that they will play on the ATV2.
I am hopeful that Apple won't do anything tricky to complicate matters (but I wouldn't be surprised if they do) so that the revision will only require putting the AC3 tracks into an appropriate Quicktime container.
Right now, there is a two-step process to get AC3 into .mov files for playback in QuickTime (first .avi, then "Save As" in QuickTime Pro as a .mov) that will play back in 5.1 on a Mac connected to a receiver by an optical cable using Quicktime, VLC or Front Row, plus installation of Perian and a small mod to a plist file. However, these files are rejected by the ATV, even though iTunes accepts them.