With the custom anamorphic setting checked, you encode the entire movie frame at the correct aspect ratio at 1280x720. That is at a 16x9 aspect ratio, but the original aspect ratio, whatever it is, is preserved, because the pixels are not square. They are rectangles, in a ratio necessary to preserve the original aspect ratio of the movie. For a widescreen movie, your actual horizontal resolution with always be 1280, but instead of square pixels you will have rectangular pixels, which your AppleTV or Mac Mini will unpack as 1680 or so. This trick really does not improve horizontal resolution, it is just a trick. However, you do end up with a real improvement in vertical resolution.
If you set horizontal resolution to 1280 and allow the vertical resolution to float, on widescreen movies, that resolution can be as low as 550 pixels. With custom anamorphic, you always get 720.
An anamorphic encode with have 921,600 pixels, whereas a non-anamorphic encode will have about 704,000 pixels. That is a 30% increase in pixels in vertical resolution, and my understanding is that vertical resolution has a bigger impact on perception of visual clarity.
I know that resolution is not everything with respect to encodes, but all this being equal, more pixels are better.
Many thanks. I'll give this a whirl on my next encode.