Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
First, use the Handbrake High Profile preset to encode your DVDs. Never, ever set RF to 0, as it creates a file several times larger than the original. High Profile will retain 99% of the DVD quality while resulting in a manageable file size.

As for your picture quality issues, Handbrake has decomb and deinterelace filters built in. I'm not near a computer at the moment, so I can't check, but I believe they are in the Picture tab of the Handbrake GUI. Make sure both of them are enabled, and this should solve your problem.
 
You are wasting your time.

When encoding you are never going to achieve lossless, simple as that. If you really feel like you can see a difference no matter what you do - then look for ways to play the .iso straight from the appletv. I don't have an appletv2, only part 1, which can be hacked to do this. not 100% sure about the state of appletv2 hacks, sorry.

I have a 60" panny plasma and with the appletv preset and RF set to 19 and I can tell a slight difference if doing an A/B comparison. But to be honest after a few seconds my eyes adjust fine and I could care less. Especially with the convenience of having everything at once and no scratched disks nor long ass menu's to fast forward through.

One last thing is how is your appletv hooked up to your tv? Maybe the upscaler in your tv isn't liking something for some reason? Shrug.

Although at the end of the day, SD looks terrible to me for the most part and I stopped ripping DVD and only rip Blu Ray now.
 
The reason your encodes are bigger than your ISO is that DVDs aren't uncompressed: DVD video is compressed using mpeg2. So, as omni says, you're never going to get a lossless copy of your movie no matter what you do.

Most people on the Handbrake forums seem to agree that RF19-19.25 is indistinguishable from the DVD source, so I'd go with the Apple TV 2 preset and adjust the RF to one of those if you really feel you can tell an encode with the stock ATV2 preset (RF20) from the DVD. It might help for you to do a few test encodes (one chapter of a DVD you know well) using the stock ATV2 preset, ATV2 preset with RF19 and ATV2 preset with RF19.25 and do a proper a/b/x comparison:

  • Play 1 version (A)
  • Play other version (B)
  • Play one of A or B (X) and note down which of A or B you think X is

Do it with someone else starting the video for you so you don't know which is which. You could do the same for the source DVD vs your encodes too. If you can't reliably ABX your encode from the source DVD, then the encode is fine.
 
I need to learn how to do the rest of the settings. I don't seem to be able to select Apple TV2 mode or High Profile mode and then lower the RF Value so I think I need to configure all the settings manually and save that as a favourite if possible.

If you are encoding for AppleTV 2, start with that preset and then change the RF value. The preset will no longer be highlighted as the RF setting itself is part of the preset. Everything else you can leave alone. You can save this as a custom preset, but it's pretty easy to start from the existing preset and change the RF each time.
 
At RF 10 I can tell no difference in the contrast between the DVD but the motion seems a little jerky only fractionally. RF 11 is nearly as good contrast wise and I notice no motion problems so I think will settle at 11. I have no interlacing problems either at these lower values.

To be honest it's all about finding what's right for your setup and eyes, so if you're happy with your chosen settings that's the main thing.

I need to learn how to do the rest of the settings. I don't seem to be able to select Apple TV2 mode or High Profile mode and then lower the RF Value so I think I need to configure all the settings manually and save that as a favourite if possible.

There should be a slider in the HB window adjacent to where the RF value is shown: if you move it right it'll raise the RF and moving it left will lower it, even if you've selected a preset as your start point IIRC. Once you have it all set up you can save your settings as a custom preset.

I also need to find out how to get the best sound. Some of the DVDs I want to keep are Music.

You haven't really got a lot of options there re: Apple TV - you can either have 5.1 surround by passing through the AC3 from the DVD, or you can have aac stereo sound. The Apple TV presets in HB set this up for you automatically so it should be sorted for you if you use one of those presets as a starting point.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.