After trying quite a few different methods, these settings below using the most current version of Handbrake (Version 0.9.2 (2008021900)) give me the best possible picture with 5.1 sound. The only exception is if the main feature/movie size is in the 2 1/2 hours or longer in length, I may have to reduce the average bitrate to 3000 or if after doing so, it's still larger than 4 GB where the bit rate is that low, I will split it into two titles (e.g Part I, Part II, etc.). I've only had to do this with 4 movies out of over 170 (all 3 LOTR extended editions and Amadeus with parts I & II for each feature).
Of course with some older releases or lower budget stuff (music concerts, stand up comedy concerts, etc.), fast interlacing may be needed, so this will slow things down a bit more. Every once in a great while I'll also need to manually futz with the picture settings as the anamorphic strict won't be ideal, but for 85-90% of the features, my default settings below do the trick.
I'm not terribly concerned with file size as I want video quality as close the original source as possible, but from my experience (and other feedback I've read), Handbrake crashes on any MP4s over 4 GB and the Apple TV has issues with them as well. My average feature size is approximately 3 GB and these settings do take some time to encode on my G4 Dual 1GHz MDD (6 1/2 to 9 hrs.), but I can't see the sense in making low-res versions as my main use is for in-home viewing on 42" screens and up... -TD
Format: MP4
Codecs: AVC/H.264 Video / AC3 Audio
Framerate: Same as source
Encoder: x264
Quality: Average bitrate (kbps): 3500
Picture settings: Anamorphic: strict