Hey guys, 1000000000000000x thank you for all of your guidance and advice, wouldn't have been able to do any of this without your generous sharing....but just wanted to type out a post to collect all the steps I had to do, in one place, since I was clicking, scrolling, searching to piece everything together.
Just wanted to let you know what I have been able to build, might want to update as well.
I have 10.15.6 running on my
Mac Pro 5,1 12 core 3.66 W
with 64 GB Ram 1333mhz
RX 5700 xt GPU
Booting from NVMe SSD via PCIE slot
First, you have to be willing to get your hands dirty.
In order to get a GPU like the 5700 running, you have to do a PIXLAS mod...this involves taking the Power core out of the machine, purchasing a cable to then splice the wires coming out of the power core...this is so you will bypass the thin wiring on the motherboard to the Mini 6 pin power ports....you go straight from the power core to the GPU which is 6 Pin and a 8 pin....if you just YouTube PIXLAS you will see a great video on how to do it and where to order tools and wire needed.
But to get that 5700 card to work on the machine, you have to be running 10.15.... You need to turn off SIP on you 5,1running Mojave, while in recovery mode, open terminal:
csrutil disable
restart computer to take effect.
Then you need to open terminal and enter this command, can be in normal boot up mode, not recovery mode.
sudo nvram boot-args="-no_compat_check"
Now, you have to get your hands on a compatible computer that is running Catalina 10.15 already....get an SSD or HDD and hook it up to the computer via external drive. boot into recovery mode, formate drive to apfs and then install Catalina 10.15 on that external drive....then take that drive and stick into the Mac Pro 5,1. Your Mac will now be running Catalina 10.15. (I was running Catalina even with my old RX580), you can now use time machine to rebuild your system in 10.15.
You can now take out your GPU and slap in the rx 5700 xt, plug in the cables coming from the PIXLAS mod and you will be good to go!
I also used Carbon Clone Copy to clone my SSD on to the MNVe drive and that is how I am booting from NVMe drive (note: you have to have BootROM 140.0.0.0.0 version, without this version your 5,1 will not recognize NVMe drives.
Now the great simple task of getting Boot screen and native OS updates through Mac app store (since officially 5,1 is not supported passed 10.14) you need to download OPENCORE. here is a video that shows you the simple process to do that, with download package in comments.
Note: when I mounted my EFI, it did not have an EFI folder in it, so I just dragged the whole EFI folder from the download.
Aslo, you can re-enable SIP as well, VIA terminal while in RECOVERY mode.
csrutil disable
or
csrutil enable
I hope this helps out novice tech guys like myself, please let me know if I laid out anything incorrectly, to my knowledge, this is what I did in total to get my rig up and running.
also if you are interested in this mod but think its to complicated, its not....some my just stay with the RX 580 bc its plug and play but check out these bench scores when compared to the RX 5700 XT in a 5,1 Mac Pro set up.
https://barefeats.com/cmp-5700-xt-vs-other-gpus.html