x264 vbv bug

[paol]

I recently compiled Handbrake svn to replace VisualHub for some CRF-goodness

However, as is stated at the Handbrake forums, there seems to be a bug that makes some frames blocky using vbv. There is very little info but I have done several tests, its worse with HD sources ofcourse since the bitrate is higher. Anyway, its preventing me from making the great encodes I like.

All this testing showed that ATV can handle very high bitrates with no problem, 13 - 15 Mb/s is fine if CABAC is off, maybe even 20 (using VLC to check).

Anyway, the happy news is this thread in case you haven't noticed: http://forum.doom9.org/showthread.php?t=140326&vbv+settings&page=3

Hopefully they will solve this soon. I have actually been encoding without vbv in the mean time, the ATV can usually handle it, even streaming Casino Royal however has scenes with more than 30 Mb/s which the ATV shows in cool slow motion

I hope this problem isn't discussed in another thread, when I search google for vbv, almost nothing interesting turns up.

 

[paol]

Fixed!

Ok, the bug is fixed in the latest revision of x264 (949)! I am very excited since this has been very apparent in many of my TV-shows, although the ATV can handle them without vbv even in 720p, but I like to be safe And it will be great for the HD movies.

Any chance of implementing the latest build of x264 in HandBrake soon Dynaflash?

On an unrelated note. I tested Casino Royale 720p with vbv-maxrate = 10 000, vbv-bufsize = 12 500. CABAC off. Streaming was perfect even when the bitrate for long periods of time (10 sec or so) was 10 Mb/s with several peaks well above 15 Mb/s, probably higher but VLC polls to seldom. Without vbv the same clip will be > 20 Mb/s with peaks of 30 Mb/s which my ATV will show with dropped frames as I said above.

 

[dynaflash]

Any chance of implementing the latest build of x264 in HandBrake soon Dynaflash?

Well, yes and no. I am running tests now in our svn, its very nice. But tbh am waiting for psy-rdo before officially updating the HB svn. Worry not. HB dev's pretty much exclusively use x264 for the vast majority of encodes and the x264 git repository and irc channels are monitored daily

Keep an eye out for the next dev snapshot.

On an unrelated note. I tested Casino Royale 720p with vbv-maxrate = 10 000, vbv-bufsize = 12 500. CABAC off. Streaming was perfect even when the bitrate for long periods of time (10 sec or so) was 10 Mb/s with several peaks well above 15 Mb/s, probably higher but VLC polls to seldom. Without vbv the same clip will be > 20 Mb/s with peaks of 30 Mb/s which my ATV will show with dropped frames as I said above.

At what crf setting ? VLC polls on the second according to the vlc devs (we keep an eye on them as well). As well, without cabac the atv really can handle an awful lot. Personally I keep it a bit lower bitrate/vbv wise and use cabac. But that is totally personal preference.

 

[paol]

compiled!

Ok, I really don't know what I am doing. But I compiled the latest svn of HB (1653) and before I exchanged the x264 folder with release 947. I don't know if that's how you do it, but it compiled and a test encode that used to show severe vbv errors, now encoded perfectly

If it hadn't been for the HB forums and the thread for advanced settings for ATV (http://forum.handbrake.fr/viewtopic.php?f=7&t=5129), I would never have gotten this far. Handbrake is excellent and makes it all fun and I enjoy exploring the limits of the ATV which seems to be higher than I had imagined

Dynaflash:
At what crf setting ? VLC polls on the second according to the vlc devs (we keep an eye on them as well). As well, without cabac the atv really can handle an awful lot. Personally I keep it a bit lower bitrate/vbv wise and use cabac. But that is totally personal preference.

I use CRF 64 % with these settings: bframes=6:ref=3:mixed-refs=1:subq=6:me=umh:no-fast-pskip=1:brdo=1:direct=auto:bime=1:merange=24:cabac=0

I have done very few tests with bitrate limits, but intend to do more testing CABAC is definitely an option that I intend to use. For now I have it as standard for DVD sources (since they have low bitrates anyway), but not for HD. With the limited research I have done, it seems to me that the bitrate can go alot higher without CABAC and thus providing greater quality. With my few testfiles, CABAC has saved me less than 10 % in file size and reduced maximum bitrate alot more than 10 %.

Lowering CRF to 62 and 60 % did not lower the very high bitrate in the fighting scene in the beginning of Casino Royale very much, it stayed above 20 Mb/s but the ATV could almost play it at 60 %.

I am very happy with my ATV and I have found that I use it a lot more than I anticipated when I bought it. Reencoding everything for ATV seemed like a pain but has not been very difficult at all and is quite fast. Or was before I discovered the advanced settings

 

[dynaflash]

Ok, I really don't know what I am doing. But I compiled the latest svn of HB (1653) and before I exchanged the x264 folder with release 947. I don't know if that's how you do it, but it compiled and a test encode that used to show severe vbv errors, now encoded perfectly

Well, its not how you do it, though it happens to work between the two rev's you used, since we do not currently patch x264, though at any given time we might in which case it would not work. So, you got lucky.

Reencoding everything for ATV seemed like a pain but has not been very difficult at all and is quite fast. Or was before I discovered the advanced settings

yeah, alot of the advanced options for x264 goodness do come at the expense of increased encoding time. There seems to really be few free lunches.

 

[erise]

This is great news! I've been unable to encode with VBV ever since HandBrake 0.9.2 in February because of the blocky frames. My iPod 5G skips without VBV, so I really need it. Hopefully a new HandBrake Snapshot or SVN with this version of x264 will be released soon. HandBrake's the only program I know of that can properly encode Futurama.

Paol, any chance you could upload a copy of your patched HandBrake somewhere so I can try it out?

 

[paol]

Dynaflash:
So, you got lucky

I like that Better not compile any new versions then until you do your update with psy-rdo then. But frankly, I don't need to. My latest build of HB does everything I could ever ask for Except letting me encode just a portion of an mkv (that would be for testing purposes, encoding Casino Royale takes a great deal of hours, just to check the first scenes).

Dynaflash:
yeah, alot of the advanced options for x264 goodness do come at the expense of increased encoding time. There seems to really be few free lunches.

I bet you recognized the settings Its not free, but I have two iMacs that can encode at night, about 6-7 fps for 720p, so its not a problem and definitely worth it

erise:
Paol, any chance you could upload a copy of your patched HandBrake somewhere so I can try it out?

Getting rid of the vbv-bug is a lifesaver for me too HB is about 15 - 20 Mb I think, maybe I could just mail it to you when I get home from work?

 

[erise]

Getting rid of the vbv-bug is a lifesaver for me too HB is about 15 - 20 Mb I think, maybe I could just mail it to you when I get home from work?

That would be great! If you zip it up first, the filesize reduces to only about 4MB. You can send it to erise (at) u.washington.edu

 

[paol]

That would be great! If you zip it up first, the filesize reduces to only about 4MB. You can send it to erise (at) u.washington.edu

No problem, I am happy to help you out. I am very excited about this myself. It will take another nine hours before I am back home though, but you will get it then, zipped and ready Just remember that neither I nor the Handbrake devs can support this version if you have any trouble.

 

[dynaflash]

Just remember that neither I nor the Handbrake devs can support this version if you have any trouble.

Ditto. It is a conundrum regarding open source. Though it solves your current issues with vbv to some extent (there has already been a couple more tweaks to x264 since you did that) it very likely could introduce other issues. many days there are commits to our svn that fix bugs introduced earlier in the day. So whenever using an svn build ymmv. Big time.