Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
So, update:

I have not yet had a chance to troubleshoot by connecting my MBP to the Samsung and playing the same movie (need to locate some cables).

However, in the meantime -- I received a Roku XS for Christmas. The same movie, played on the Roku (over a G network no less, where the ATV was on N), is noticeably smoother than when played through the ATV. So, I'm pretty much convinced it's not the way I'm ripping these, and the TVs seem to be capable. Guess maybe I just need to wait until Apple has a 1080p native AppleTV, or something.
 
Sounds to me like your converting them at the wrong frame rate. Don't let handbrake guess the framerate, tell it what it is..see if that makes a difference.
 
same issue here

I have a Mac Mini Server running iTunes to ATV2 over Gigabit Ethernet with files created from MKV>Handbrake ATV2 conversion. It is displayed on a Samsung 60" LCD with 720MHZ clear motion and my video is also jittery at times. Seems to also happen on downloaded HD content from iTunes, so it must be something to do with the TV setting.

So far no solution!
 
I have a Mac Mini Server running iTunes to ATV2 over Gigabit Ethernet with files created from MKV>Handbrake ATV2 conversion. It is displayed on a Samsung 60" LCD with 720MHZ clear motion and my video is also jittery at times. Seems to also happen on downloaded HD content from iTunes, so it must be something to do with the TV setting.

So far no solution!

I bet your sammy is set to smooth the video which set's it's refresh rate incorrectly to match the media atv2 plays. Let me see if I can find the info regarding that. Ok it's the smooth motion, you need to set it to off for your ATV2 input.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.