The PS3 can play file of any size, but as you've discovered the problem is getting the files on to the PS3. The PS3 will only read drives/memory sticks that are formated using FAT32, which has a 4Gb file size limit.
If you running a Windows system then you could use
RedKawa Fileserver to copy the files to your PS3 over your network.
1. Download and install
RedKawa Fileserver
2. Run RedKawa and make a note of the web address it gives you at the top of the window, for me it was 192.168.2.100:8192
3. Press the launch File Server Button, this should open a webpage with a list of drives on your computer.
4. Press the Settings button at the bottom of the webpage.
5. Under the Approved Download Directories section, add the location of your movie files. eg mine was C:\Movies\
6. Now goto your PS3 and start the web browser.
7. Enter the address that you noted from step 2
8. You should have the same web page that you had on your PC.
9. Navigate to the movie you want to transfer to the PS3.
10. If you press X on the movie it will stream to the PS3 and play.
11. However we want to copy it to the PS3 so press Triangle.
12. Go down to the file menu and press X.
13. Now go down to save target and press X
14. Now wait this can take a very long time if your file is large..........
15. Select the destination Hard Disk and press X.
16. The file is now being downloaded to your PS3 hard drive.
17. Mission Complete


