In 4.1, the Apple Devices HD (best quality) is not multi-pass... Or single for that matter, nor is it 10kbs as I assumed by looking at the output files. It's apparently set to "automatic" on both of those critical settings, which means sometimes it's multi pass, sometimes single (which can use QuicksSync on i7s).
I just did a quick test comparing Compressor 4.0.7 (yes, you have 4.1, so results may vary) against Handbrake.
Computer
A temporary 2012 Mac Mini with 2.5Ghz 2-core (4 with HT) i5 that I am running temporarily until my new Mac Pro comes in. My old Mac Pro died (the long story).
Source
A 30 fps 1280x720 progressive video, exactly one minute long, that I rendered out of After Effects a few years ago. It's a bunch of 3D butterflies flying around in front of some stock footage. I made it when I was building this tutorial for VTC
http://www.vtc.com/products/Adobe-After-Effects-6.5-Advanced-tutorials.htm.
Results in Compressor 4.0.7
I used the HD for Apple Devices (10 Mbps) preset unmodified
Encoding time was 2 minutes, 27 seconds
CPU utilization was 172%
Final bitrate is 9.38 Mbits/sec
Dropped my 30 fps source to 29.97 fps
Results in Handbrake 0.9.9
I used the Apple TV 3 preset unmodified
Encoding time was 1 minute, 2 seconds
CPU utilization was 390%
Final bitrate is 1.67 Mbits/sec
Maintained my 30 fps source as 30 fps
So faster encode and smaller file. However, the smaller file is of lower quality, which is to be expected on one level. There is an easy remedy for that, but let's take a look at just how bad. Interesting note: Compressor had shifted the frames off by one. I had to bump the encoded version by one frame to properly line-up the action. Dunno if this is because of the conversion to 29.97 or not.
Darker is good; lighter is bad. You can see the characteristics of the different approaches to quantization of data. On further experimentation, it looks like a Constant Quality setting of RF10 in Handbrake yields results similar in quality to what I am seeing with that particular preset in Compressor.
----------
Just finished running my master through HandBrake. I used the "Normal" preset which is at your suggested RF=20.
RF20 is a quantization setting, not a bit rate. What you are telling the transcoder to do is disregard bit rate, but instead always compress every frame and every GOP (group of pictures) using one compression setting. The bit rate will go up if there's action or detail and will go down if nothing much is happening.
RF20 is great for compressing movies for viewing on an Apple TV. In fact, to my eye, it's overkill and I usually drop down to RF22 or even RF23. However, if you want higher quality, quality that perhaps matches what you are seeing with that particular preset in Compressor, raise the RF to 10. You will get similarly large file results and the encoding should take little to no more actual time to complete.
----------
It's crazy that they beefed up the graphics cards so much for engineering, 3D graphics and scientific buyers (I'm sure they have deeper pockets) but don't have an optimization for video editing. Say what you want about the older model, but one great thing was if you found a card that did something you didn't have, you could add it.
I have a gut feeling that the Handbrake coders/contributors are salivating over the power of the D700s and are probably already figuring out a way to use them. They seem to be fanatical about performance, seeing as how they always fully saturate the CPU cores.
----------
As such, it created a file size of only 113 MB, as opposed to 600+ in the Compressor versions, and over 8GB in the master.
If you are compressing a source file that large, then the time it takes to get 8GB of data into the transcoder could become part of the problem. I was working off an SSD for my tests and with only a 360MB source file.
I should have asked from the very beginning, but what is the purpose of the compressed version of this video? Is it for archival purposes or for some sort of distribution. That will determine what bit rate or RF setting to use. Also, it will impact the audio bit rate you choose.