Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

How can I achieve 720p Handbrake encodes that don't stutter?

N

Number6

macrumors member
Original poster
Jul 29, 2008
47
1
Is there anybody here who has no problems with HD files that conform to Apple's specs for the Apple TV? I just finished a long encoding and tagging session only to be totally frustrated. Two out of three files stutter (or drop frames) during playback on the Apple TV :(

At first I tried steaming them over my n-network wirelessly but that was totally unwatchable. After syncing them to the Apple TV things improved a little bit but it still seemed like the movie freezes and picks up speed again every once in a while. That is just not good enough.

These files play just fine in Quicktime and iTunes and I encoded them with Dynaflash's advanced options which I added to Handbrake's (Version 0.9.4 x86_64 (2009112300) Apple TV setting. The advanced options read like this:

ref=3:mixed-refs=1:bframes=3:me=hex:subq=7:b-adapt=2:8x8dct=1:weightb=0:trellis=0:weightp=0

Constant quality is at 60.78% or RF 20 and I bumped the resolution to 1280x whatever the aspect ratio requires. Is there anything wrong with these settings?
I went with the standard Apple TV settings in Handbrake at first but if I bump up the resolution the problems are the same and the 960x540 resolution doesn't cut it. The framerate on all these files is 23,98 so a real 720p resolution should not be problem.

The crazy thing is that one file works and plays without stuttering (even over the network!) so what can I do to make all files come out that way?

I've had these problems ever since I started encoding HD video and I find myself re-encoding the same movies over and over and I'm sick of it. The iTunes HD movies look great and have similar specs and I want my collection in that quality, but it seems unachievable. My WDTV HD Media Player plays nearly everything I throw at it but the interface sucks so hard I would rather live with 720p video if I could get it to work.

Any ideas are greatly appreciated...
 
I'm sure dynaflash will comment here soon enough, but you should remember that he syncs all of his content which gives him a little more flexibility on what the AppleTV can handle (eliminating the streaming overhead). My advice would be to download one of the nightly builds of Handbrake (lots of fixes for things you'll likely come across in HD sources) and try the stock AppleTV preset, modified to use 1280 width. The updated preset uses vbv limits to help prevent the bitrate spikes that cause stuttering on the AppleTV.

The only dropped frames that I see anymore are in the first few seconds as the AppleTV starts to buffer the content. If the problem persists, you could try increasing the RF value to 21-23 which is more in range with the values recommended over on the Handbrake forums for AppleTV 720p encodes (although I use RF20 without issue).
 
That was quick! Thank you, I will try it tonight and post results.
 
Number6 said:
That was quick! Thank you, I will try it tonight and post results.
Click to expand...
Sorry for all the edits to my original post; I should know better than to read Macrumors while sitting in a meeting. :cool:

There is a lot of info regarding Bluray/HDDVD sources and Handbrake in the thread linked below, although I admit the info I provided in the first post is a little dated. One of these days I'll get around to updating it.

http://forum.handbrake.fr/viewtopic.php?f=14&t=9258
 
Increasing the resolution is a sure way to make it stutter. Use the original image size that Handbrake suggests, Strict Anamorphic in Picture Settings. Anything higher is a waste of CPU time. If your originals have bad quality, add a noise filter. You can try turning CABAC off, that is quite hard on playback, and you can try changing deblocking to (-1, -1) or (-2, -2); both options lose a bit of image quality but make it a lot easier on the decoder. And you might try what happens with "average bitrate" instead of "constant quality". Average bitrate would make every bit of the movie equally hard or easy to decode; "constant quality" would use more bits for critical parts of the movie to keep the quality up, but that might be what produces stuttering.
 
gnasher729 said:
Increasing the resolution is a sure way to make it stutter. Use the original image size that Handbrake suggests, Strict Anamorphic in Picture Settings. Anything higher is a waste of CPU time. If your originals have bad quality, add a noise filter. You can try turning CABAC off, that is quite hard on playback, and you can try changing deblocking to (-1, -1) or (-2, -2); both options lose a bit of image quality but make it a lot easier on the decoder. And you might try what happens with "average bitrate" instead of "constant quality". Average bitrate would make every bit of the movie equally hard or easy to decode; "constant quality" would use more bits for critical parts of the movie to keep the quality up, but that might be what produces stuttering.
Click to expand...

The only thing that I really agree with you is disabling CABAC as it is likely the "straw that broke the camel's back" in this instance. The updated AppleTV preset disables CABAC and enforces vbv limits to prevent bitrate spikes larger than what the AppleTV can handle (as shown through testing). The AppleTV preset defaults to 960x540 to ensure maximum compatibility, but you can create 1280x720 encodes as long as you're aware of the playback limitations of the AppleTV.

That said, we're all entitled to our own opinions on what settings produce the best output. All that matters is that the results look good to you.
 
I know others here disagree, but my stuttering problem on certain files (e.g. Gladiator) would not go away no matter how I reduced the resolution of the encode. I was reducing a 1080p MKV to DVD resolution and it was still stuttering on the AppleTV. Then I changed the frame rate from "Same As Source" to the hard setting "23.98". BAM! Everything was perfect -- no stutter. I could even boost the resolution using the custom anamorphic setting to something like 1723x720 and still no stutter. It was beautiful. So, now, I first check in VLC to find out the actual fps of the MKV file and hard select that frame rate in handbrake as oppose to allowing handbrake to determine and select the frame rate for me. I think I have read that there is something wrong with some MKV's made by MakeMKV.
 
I have encoded a ton of Blu Rays through Handbrake. Every once in a while, one will stutter in the manner you describe. I simply re-encode the same rip using the same specs and so far not one has failed to come out perfectly on the second (or very occasionally third) try. It's a ballache to have to run the encode more than once, but it's either that or not have the movie in the library.

I have no idea why some come out that way. The only correlation I can infer is that the stuttering happens when I try to do anything else with my Mac while it's chewing on an HD encode. This makes no sense, but since I have made it SOP to avoid touching my Mac while it's at work (which includes not watching anything on my Apple TV that may be streamed) I have had 100% stutter-free results. If nothing else, this allows me to justify to myself the planned upgrade to an i7 iMac...

FWIW, I use the Apple TV preset, but adjust the picture settings so that anamorphic is set to "None" and the resolution is fixed at 1280 x xxx. That's just a personal preference, and I doubt it has anything to do with stuttering encodes.
 
These are the advanced options I use:-
Capture.JPG
All of my HD video is encoded at 1280x720 and has a bitrate of around 6-6.5Mbps, I stream all of my content over a wired connection and my ATV never has a problem. I curently use HB 0.9.4svn3154 but have never had problems even using older versions.
 
As others have mentioned, and also shown in roidy's screenshot, disabling CABAC is the key to this one. Although not officially supported as per Apple's ATV specs page I think it is safe(ish) to use on SD material but I would say it was a no-go on an HD source.
Your other advanced settings look pretty close to what I use and I have no problems with HD playback, with CABAC off.
 
First of all - thanks to all the people who replied. Just finished my re-encode and I am streaming (not syncing!) a 720p HD file with no stuttering to my Apple TV right now :D

The first thing I tried was what NightStorm suggested and got the latest nightly build of Handbrake. Using the new Apple TV preset I raised the 960 resolution to 1280 and that's it. The resulting file looks very sharp and finally there is no S-T-U-T-T-E-R!

The only thing that's not quite iTunes store HD quality are complex shades of color (mostly in backgrounds, sometimes in faces) which seem to become "busy" or having some kind of solarization effect. Hard to describe, might be because I am dealing with material that has been compressed before?

In the last column of roidy's settings there are some different values but I really don't know what they do. Maybe these could help with the "moving walls"?

Anyway, I will try the other failed encodes now, experiment a little bit and report back.
 
Number6 said:
First of all - thanks to all the people who replied. Just finished my re-encode and I am streaming (not syncing!) a 720p HD file with no stuttering to my Apple TV right now :D

The first thing I tried was what NightStorm suggested and got the latest nightly build of Handbrake. Using the new Apple TV preset I raised the 960 resolution to 1280 and that's it. The resulting file looks very sharp and finally there is no S-T-U-T-T-E-R!
Click to expand...
Glad to hear that worked for you. I've been using those settings for 50+ Bluray/HDDVD encodes with no issues so far.

The only thing that's not quite iTunes store HD quality are complex shades of color (mostly in backgrounds, sometimes in faces) which seem to become "busy" or having some kind of solarization effect. Hard to describe, might be because I am dealing with material that has been compressed before?
Click to expand...
You're on the right track here... Apple/"the studios" have access to much higher quality sources when they encode for sale in the iTunes store.
 
Number6 said:
The only thing that's not quite iTunes store HD quality are complex shades of color (mostly in backgrounds, sometimes in faces) which seem to become "busy" or having some kind of solarization effect. Hard to describe, might be because I am dealing with material that has been compressed before?

In the last column of roidy's settings there are some different values but I really don't know what they do. Maybe these could help with the "moving walls"?
Click to expand...

Yep if you look at my screen shot you'll see I use both the 'No Fast-P Skip' and 'No DCT-Decimate' options and a deblocking value of -1,-1.

First the 'No Fast-P Skip' option is the one you want to stop the problem your having, namely the color banding(solarization effect). Don't let anyone tell you that color banding is because iTunes uses higher quality source because it isn't, I've seen the same problem on quite a few iTunes purchased encodes. If there is no color banding on your source material then there is no reason it should be on your encoded version.

Second the 'No DCT-Decimate' option will prevent the encoder from destroying subtle detail in the source material like filmgrain.

Third, the -1,-1 deblocking values are because I think Handbrake makes your encodes too soft and blurry with the default values.

The only downside to these options is a slightly longer encode time. Give them a try and see.
 
Wirelessly posted (Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16)

Can someone tell us where to get the latest build of handbrake? I went to the supposed "snapshot" page, but no download was available. Thanks.
 
stevol said:
Wirelessly posted (Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16)

Can someone tell us where to get the latest build of handbrake? I went to the supposed "snapshot" page, but no download was available. Thanks.
Click to expand...

Handbrake switched to nightly builds... There is a sticky in the handbrake forums with all the info you need.
 
Wirelessly posted (Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16)

Does the more recent nightly build(s) have any significant changes or improvements to the ATV preset? Would love to have a built-in preset for mkv from blu-ray to ATV (like an ATV 720p preset). Then we can relax on these forums knowing that the issues we are having are resolved (for now) and that the frustrating process has now become a piece of cake!
 
The nightly builds aren't designed to address the presets, but to enhance code through bug and performance fixes.

You will have to experiment with the detailed settings on your own to identify a set that looks good to your eye, is optimal for your source material, and is optimal for whatever hardware you have on hand to encode with.
 
brentsg said:
The nightly builds aren't designed to address the presets, but to enhance code through bug and performance fixes.

You will have to experiment with the detailed settings on your own to identify a set that looks good to your eye, is optimal for your source material, and is optimal for whatever hardware you have on hand to encode with.
Click to expand...

There are some changes to presets from .9.4 to the nightlies. For example, the AppleTV preset now uses vbv limits to prevent bitrate spikes that cause issues with smooth playback on the device, especially for HD sources.
 
NightStorm said:
There are some changes to presets from .9.4 to the nightlies. For example, the AppleTV preset now uses vbv limits to prevent bitrate spikes that cause issues with smooth playback on the device, especially for HD sources.
Click to expand...

Good to know, thanks.
 
roidy said:
Third, the -1,-1 deblocking values are because I think Handbrake makes your encodes too soft and blurry with the default values.
Click to expand...

People will argue whether stronger or weaker deblocking looks better, but deblocking needs to be done during playback, so less deblocking (-1, -1 or even -2, -2 instead of 0, 0) will use less CPU time during playback and therefore reduce stuttering.
 
I agree things like deblocking values are subjective, what I think looks good may not look good to everyone. However when I was testing setting to get the perfect encode I tried a wide range of deblocking values from -4,-4 right upto 4,4 and the ATV never had a problem playing any of them so while deblocking might have some effect on cpu usage during playback I doubt it would be to the extent as to cause any playback problems.
 
Finished a couple of encodes with roidy's deblocking and anti-color banding settings and here are my (mixed) results. The movie that was finally stutter-free but still had "busy" walls sometimes (see post #11) was redone and showed very slight improvement in the walls, but the difference was so small that it could've been subjective. An A-B comparison was difficult since there was always a little time gap but in the end the results were very similar. File size and encoding time were nearly identical as well.

I checked the original mkv (a rather large 1080p encode) in VLC and the artifacts were definitely not there. So my encoding should be the culprit. I wonder if I increase the file size would the encode profit from it? Going from 13,23 GB to 2,63 GB, there must be some visible loss. The iTunes store file of "Quantum of Solace" is 3,8 GB, looks great and is roughly 20 minutes shorter in running time. I have a Drobo and space is really not much of an issue. I would even go for 6 GB if it yielded better results.

Than I checked another file that was very interesting. A VERY grainy 70's movie that originated from a 6,59 GB mkv and came out at 2,57 GB with only 88 minutes runtime. I started streaming the movie and after a couple of minutes I encountered the dreaded freeze and stutter again. Not as severe as before but still there. So I decided to sync the file and continued watching. There was no more stutter but the most amazing part was that the grain was intact! And I'm talking a lot of grain and fog here. No compression artifacts anywhere to be seen and the movie looked like it should. I never encoded this before but surely roidy's settings must have been a factor here.

This result is sure worth syncing and maybe if I had a wired network that wouldn't even be necessary. Anyway I am at loss on how to not just create bigger files but better ones. I know that I can set a desired file size in Handbrake but that does not seem very scientific. Which size is "best" for which movie? Most Handbrake users seem to advise on using constant quality.

If anyone has any further thoughts please share.
 
Umm... The fact that your original 1080p source file is only 13.23Gb means it's already been heaverly compressed to start with so getting a good re-encode from it may be difficult, plus 2.63Gb for a encoded HD movie seems very small to me. Use 'Get info' in iTunes to see what the actual bitrate for the file is. iTunes purchased HD content usually has a bitrate of about 4500kbps, all of my HD encodes have a bitrate of about 6000-6500kbps. Looking at your file size I think your bitrate maybe too low. Post all the settings your using.
 
roidy said:
Umm... The fact that your original 1080p source file is only 13.23Gb means it's already been heaverly compressed to start with so getting a good re-encode from it may be difficult, plus 2.63Gb for a encoded HD movie seems very small to me. Use 'Get info' in iTunes to see what the actual bitrate for the file is. iTunes purchased HD content usually has a bitrate of about 4500kbps, all of my HD encodes have a bitrate of about 6000-6500kbps. Looking at your file size I think your bitrate maybe too low. Post all the settings your using.
Click to expand...

I have quite a few HDDVDs that, once extracted wtih eac3to and remuxed using tsmuxer, are approximately 13-15GB when all extras have been removed.

While 2.63GB may seem small, you've really got to trust that the x264 encoder knows what its doing when it decides how many bits to use for a particular section of video. For the longest time, I had trouble with how small my HD encodes were coming out of Handbrake (for example, my HP6 HD encode finished smaller than the digital copy included from the studio). But after doing a lot of A/B comparisons, I finally accepted that x264 knows what how to efficiently allocate/distribute bitrate without compromising quality or bloating encode size. If I had to give a range, I'd say my encodes typically finish somewhere between 2.5-4.5GB in size and look great on the AppleTV.

As stated previously, I use the stock AppleTV preset (from the nightly), with the width increased to 1280. Many devs/mods recommend increasing RF with HD sources, but I decided to keep it at 20 to try and keep my presets as similar as possible.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back