Handbrake files output to MPEG-4 should be PS3 compatible. The catch might be that it doesn't understand the .m4v extension and barfs on that. If it isn't working out of the box, you should try to simply change the extension to .mp4 and see if that will stream.
What is the error the PS3 is giving you? It does support MPEG-4, and actually supports more parts of the MPEG-4 spec than Apple does with their products.
MP4 should work, and to understand why it isn't, we need to know more about why the PS3 is rejecting them.