Polaris GPUs are only supported with 10.12.6 and newer releases, you can't run Yosemite accelerated with a AMD Radeon Pro WX5100 nor with your previous VEGA 64, the display will not work.
You need to upgrade to a supported macOS release (understand that as a version that supports the GPU) previously to installing the GPU.
High Sierra requires a Mac Pro firmware update, you Mac Pro needs to have at least MP51.0089.B00, see AboutMyMac/SystemReport/HardwareOverview/Boot ROM Version to know the current BootROM firmware version.
See the first post of the thread below to know the requirements and pitfalls of Mac Pro BootROM upgrades:
Please fully read this first post, you will probably find that you have one or more problems described into the various notes below. Mojave will only install if you have upgraded your BootROM to the current release and your Mac Pro have a Metal capable GPU*. If you are trying to install Mojave...
forums.macrumors.com
If you gonna run only up to High Sierra, you can stop the BootROM upgrades with MP51.0089.B00, if you gonna run Mojave or unsupported releases, you need to upgrade to 144.0.0.0.0.
Mojave is the most recent macOS release officially supported with a MacPro5,1.