What you really need to understand is that the encoding speed is almost completely due to the x264 encoding library that HandBrake uses. When x264 uses the gpu, so will HandBrake however don't hold your breath as x264 is largely handcrafted asm code designed to max out the cpu, preliminary tests they have done trying to shuffle stuff off to the gpu have shown little to no speed up and in some cases actually slower. Also it is correct that HandBrake can in no way utilize the elgato turbo stick.
Now that said there are several things that could bottleneck handbrake like video filters which are not always as threaded as they could be. For instance Denoise is not threaded. So turning on denoise could slow down the same encode compared to if its off. Also realize it only takes one change in the advanced options to either slow down or speed up an encode. For instance using subq=6 is much faster than subq=11 however the resulting encode may not be good from a visual quality prespective.
Comparing exact settings on the same source is the only way to get an objective comparison of speed vs. quality and you will be very hard pressed to find a faster / higher quality encoder than x264, it has come out on top in the mmsu video encoder shootout for the last several years.
Also, another note ... since hb can soak all cpu cores in most systems systems check your cpu temps, sometimes hb can cause excessive cpu temps and the os will "thermal throttle" the cpu(s) which will cause it too cool down but you are basically underclocked. I know after I took my 2006 c2d mbp apart and cleaned it out (especially the fans) I ran about 20 degree C cooler and my encoding speeds jumped a good 7-8 fps faster.