Moving away from insulting the HB forumites - news!
Some totally unscientific testing, starting from HB's AppleTV preset in SVN 2773 (is there a newer one? I don't check so often), mainly to make sure the pixel dimensions are right (nothing taller than 560px will sync, it seems)...
I turned on weighted and pyramidal b-frames, set b-frames to 6, r-frames to 4, CABAC is on, analysis = all, and automatic direct prediction - giving the following options string:
Code:
ref=4:me=umh:b-adapt=2:bframes=6:direct=auto:deblock=-1,-1:b-pyramid=1:analyse=all
And I set about chapter 18 of a Spiderman DVD5 I had lying about - the scene in which the green goblin attacks the parade.
Well, see for yourself - the linked file will sync over to my 3GS and play just fine.
Spider-man Chapter 18 Test MP4
Note: this likely won't stream in your phone's web-browser (it doesn't in mine), but save-as and feed it to iTunes and it will sync fine.
Of note - the only limit found so far is the r-frames - cranking it up to 6 produces a video that iTunes won't sync to the phone. 5 works, and we're trying 6 but with the other hard-core options turned off, on the off-chance iTunes is smart enough to be considering all the options as a whole rather than considering each one individually.
More updates to come.
[edit] - a bare-options 6 reference frames encode still won't sync, so that's something iTunes is very simply checking for.
I wonder if it's checking some internal list of iPod specs, or if it's querying the device in question asking "what can you play back?" - I have heard tell of a video settings type plist in OS 3.X - maybe someone with a jailbroken phone can edit that plist and see if iTunes trusts it?