So are you currently doing CPU or GPU rendering? (Given the 5870 I assume CPU.)2 x 3.06 Ghz 6-Core Intel Xeon
So you have no rear USB/ethernet? I'd say the first thing you need to do is figure out if your issues are hardware or software related. Do a fresh OS install on a clean HD or SSD and see if the ports work. If not, I think you can replace the motherboard for ~$500 which would be more than worth it for that machine. (If you don't want to go to the hassle, I'd be happy to buy it off you...)...my current one in on its last legs I think. I got no sound anymore and no ports are working on the back end. I wish I could explain it better with official jargon, but an artist I am, a techie I am not, lol.
Last time I looked VRay had no GPU support on the Mac and Octane is nVidia only. These posts seems relevant to your situation.
http://forums.cgsociety.org/showthread.php?t=1279522.
https://render.otoy.com/forum/viewtopic.php?f=21&t=45056
https://forums.macrumors.com/threads/mac-pro-advice-for-3d-rendering.1830543/
Granted things change quickly and I haven't been paying that much attention. But I'd guess you'll have better luck with an nVidia card in a classic Mac Pro. The AMD cards in the new Mac Pro are essentially useless.
If you are stuck on CPU rendering then you'd have to get the 12-core nMP to see any improvement over your current box, and the difference would be marginal. You can upgrade what you have to dual 3.46 Xeons for a modest improvement. You should probably look into network rendering options and offloading the workload to some cheaper Windows render nodes.
You say you aren't a techie, but if you want to push this kind of stuff you really need to become one (at least a little bit.) Otherwise I think you are going to dump a ton of $ into a new setup that won't really improve your render times one bit.