If I were to run windows 11 or 10, knowing that macos doesn't support any RTX cards, is it possible to put a 20, 30 or a 40 series card into the machine, running the latest drives on windows?
There was a time, when Nvidia CPU's have been officially supported by Apple. Those days are long gone. Back than I did have a fast 980Ti card and was very happy. In order to run those, you had to use Nvidia Web drivers that have been allowed by Apple. To my knowledge, this relationship broke apart and Apple ended any relationship with Nvidia. Since then, Metal was introduced which made the use of Nvidia GPU's impossible. When it comes to Nvidia, you have the following options:
#1 Go back in time, use an old MacOS (Maverick/ElCapitan) and use the last Web drivers. Your Power requirements have been limited. It was common to us EVGA PowerLink to stretch it a little and use a 1080Ti and the early Titan Models. This route gives a limit in terms of old MacOS and a medium Nvidia GPU of 5 years or older.
#2 Use the very latest Apple blessed Nvidia Card which was the PYN Nvidia Quadro K 5000 Mac Edition. This card is also weak compared to your mentioned cards above. However what's special about it, its the only model that Supports Metal (1) on the Nvidia side AND runs more modern MacOS software. You would still need OpenCore to have a good running system.
#3 If you don't need any MacOS on your MacPro and want to run Windows ONLY, you can install any Nvidia GPU as long as you stay underneath a certain Power limit from the PSU side of it. This is a physical constraint only.
If you want to go this route, you either use a Pixlas mod cable or you use a separate PSU just to run your GPU. Both options work to some extent and there are many examples on this forum to see how people modded their MacPro5.1 to fit their individual needs.
#1 makes sense only if you are looking to run multiple Systems (Windows & MacOS, Linux..) on the same hardware.
As of 2024 and this very late in the MP5.1 Game, I would not recommend any of the above options. It is best to use a fast AMD card such as the 5700XT or W6800 and be fine with it. The Windows only route (#3) makes no sense because you can just use any usual PC platform that most likely is a lot cheaper and more open to upgrades. Why do it?
The small market segment that needed Nvidia (Cuda core performance) passed away long time ago. Those folks have moved on around 2018. Because of this, this ancient but still great MP5.1 platform only makes sense if you still want to use a fairly new MacOS with OpenCore and the occasional Windows 10 capability for CAD/3D Print or other Apps that can't rund on MacOS or Apple Silicone. Other than that, it would make no sense.
#4 The fourth category is the Enthusiast group that is nuts like me. It's a sport. It's a classic 911 of the IT Industry. It's the last and best Mac from Steve Jobs before the bean counters captured the ship. It's individuals like me, who hang out here for 15 years straight because of the great, lovely, advanced and highly intelligent people here in this forum. The finest folks ever. The Cream of the creative crop. That's us. That's MacRumors that makes a difference.