You'll need Catalina(at least 10.15.2) and OpenCore(for boot screen and selector+FileVault support) to run the 5700 XT on the 5,1. The drivers aren't included in Mojave. I'm running the PowerColor 5700XT Red Dragon + EVGA Powerlink on mine (no pixlas mod)
One more note, the HDMI doesn't work well on the 5700XT under Catalina so use DisplayPort. I've tested this with two PowerColor cards and the reference card and all had problems.
This guide explains how to use the excellent OpenCore boot loader to install, run, and update the latest version of macOS on the MacPro5,1, resulting in a clean, unaltered operating system just like on a supported Mac.
forums.macrumors.com
Pre-configured OpenCore package at post 1314 with DRM support and full hardware acceleration
[For existing OpenCore users, you can always download my latest OpenCore package at post #1314 for update] Full HWAccel achieved now! Everything is covered in the OpenCore thread. The step by step tutorial to enable AMD hardware acceleration is now at here, but I strongly recommend you go...
forums.macrumors.com