I have a similar issue and some insights that are interesting.
Hardware details: MacPro3.1 Early 2008 with the default ATI Radeon HD 2600 and a EVGA 650TI with 6xpin connected to the logic board since 2014. The graphics work fine and are both recognised in windows and mac.
Since 10.9 (I think) have been trying to switch to Nvidia Web Driver (upgrading the system, formatting new disks with new operation systems, and so on). I have never been able to switch.
Last week bought a Gigabyte GTX1080 ITX (pascal) with 1x8pin with 2x6pin connectors into the Logic Board.
It works well in Windows, but it is not recognised in El Capitan. From readings in the Web I realised that I should upgrade to Sierra or High Sierra (unsupported systems for MacPro3.1 but possible with some tweaks and patches).
Upgraded to High Sierra with a patch that disables SIP forever. Installed Web Driver and still the same result as ever. Read some posts in forums about the SIP. So I went to Recover drive and move the kext that disables automatically SIP (/Volumes/xxxx/Library/Extensions/SIPManager.kext) to another place and enable the SIP through csrutil enable. After boot and installing the Web driver I achieve----- the same result !!!!!!
And now for the strange thing. Yesterday I brought a Macpro3.1 from a friend to install an SSD. Having this computer in my home I tried installing my GTX1080 and formatted with the same Pendrive using the same HighSierra and fowling the same procedure but from a fresh start. After installing the system it is fundamental to upgrade to the last security upgrade (if not, there is no web driver for that OS version 17G66), however the SIP must be disable to do this upgrade. After upgrading and downloading the latest Nvidia Web Driver I went to Recover Mode, move the Kext and enable SIP. Boot and install the driver, Gatekeeper will requires opening permissions to allow the installation. Finally, after booting.....everything worked out.... the GTX1080 was now recognised by the system, tested the graphics and it was OK.
Then....Made the same procedure with my computer...and at the end.... nothing.....still in default mode....
Removed the SSD disk from the other computer and installed on mine... booted from this SSD and the result was the same ...defalut mode....
Conclusion:
Similar machines with the same hardware using the same system and the result is different. How come?