I also got those errors with Metax for no apparent reason and tried the same things you did...and had trouble with 64 bit enabled mp4 (4GB+) coming out of Handbrake too....I'm not even sure its actively developed anymore.
..personally I think iFlicks murders Metax...it has a built in scraper for IMDB and TVDB (similar to what you would find in PLEX), so if you just name the file TVshowname_SXX_EXX when you set up the encode etc. it will automatically scrape, write the data and poster to file, and drop it in Itunes if you wish. You can do them in batches, eg drop a whole season onto it and it will write each episode's unique info, or edit the metadata as you wish. I find it faster, slicker, and more reliable than Metax and has some other cool little things. The few caveats are it won't query chapter names like Metax can, but I don't care about this much myself, and to write ALL possible metadata it needs to wrap the file into a .mov container (not transcode, just save the file as .mov). Again for me don't care.
The final nice thing is that it comes with a folder action, so when you drop a file into a folder, it will automatically import and tag. I use it this way now, where I'll set up Handbrake encodes to write to a watched folder. When each encode finishes, it gets dropped into the iflicks folder, which tags it, moves it to my external drive, plops it into Itunes, and on to the ATV. It's about $20 but so much time saved.