Tiger, 1.5 ghz PPC G4. I tried handbrake in took literary half a day. it look like my machine was froozen. then i tried popcorn which I started at 4am and it was still going at 12pm.
If buying a faster mac is not an option, then a hardware solution may be your best bet... Check out the turbo.264 product from elgato. I don't have one, but I have read some good things about it. On a PowerPC mac, it should be a major speed boost.