I've been messing around with Handbrake settings recently: here's my preset for anime:
AppleTV preset with the following modifications:
- Turn off anamorphic, keep it at native resolution
- Check the De-telecine box (this is assuming that the animation is at 24fps, which has always been the case for me)
- ~1500kbps constant bitrate
- x264 encoder, one-pass (two-pass makes most of my anime material look *worse*

)
The files that result from this method look nicer on my iMac than the original TS files (because DVD Player couldn't deinterlace properly if its life depended on it).
You might try out the constant quality setting to see what happens, but those setting should provide a good base for experimentation.