First, use the Handbrake High Profile preset to encode your DVDs. Never, ever set RF to 0, as it creates a file several times larger than the original. High Profile will retain 99% of the DVD quality while resulting in a manageable file size.
As for your picture quality issues, Handbrake has decomb and deinterelace filters built in. I'm not near a computer at the moment, so I can't check, but I believe they are in the Picture tab of the Handbrake GUI. Make sure both of them are enabled, and this should solve your problem.