You must install a OEM Apple graphics card to be able to update the firmware... in my case I used the original ATI 5770 apple card that came with my mid 2010 Mac Pro 5.1. Firmware update runned flawlessly and booted back to Sierra to continue with the update to High Sierra.
Installing High Sierra went well, no hickups, I did keep in the 5770 in though. After the install and finally into High Sierra, checked if everything was working ok, did a reboot, etc.., and all was ok.
So then I shutdown the Mac, removed the 5770, replaced back the GIGABYTE RX 460 4gb, hooked up my triple monitor setup as I had before in Sierra (2 x QHD 2560x1440 on HDMi and DP outputs + 1 x 2048x1152 DVi output)... and booted the system. Blank screen throughout and no login screen afterwards. So suspecting that I needed to reboot again so that the drivers would activate for the new card (I recall having to do that when I first installed this card in Sierra), I did a hard shutdown on the button and rebooted. ALL OK ! Screen showed up for the login just as before on Sierra and all 3 monitors turned on and perfectly as before.
After testing the system further, playing some games (metal versions too)... all is ok! HS + RX460 Gigabyte rocks !
I don't think a stock Apple card is really required.
I update my firmware with the 7950 Mac Edition card. Despite it's a Mac Edition card, I flashed it myself because I want to fine tune the voltage, memory clock, etc. Therefore, practically it's a flashed card, not Apple stock card, not official licensed Apple card either.