Yes, a 460 should be plug and play. If it doesn't work, then nothing you can do, 99% means the card is not supported or faulty.
Your Mac Pro is a PC, just a specific type PC. And Bootcamp Windows is normal Windows. Of course you can do the flashing in bootcamp. However, I don't think it's a good idea to do it now. At least use the card for a period of time, make sure it's normal (no need for RMA or return), and then decide if you really need that extra power (also potential extra trouble).
So, before you flash the card, you know how it perform. And you can tell if there is any difference (both good and bad) in real world after flash, do you want to keep the unlock etc.
Of course, if you don't care about the card, you just want some fun. That's it's another matter.
Just got it plugged in. Here's my situation - maybe you can give me some advice:
Removed the 5770.
Installed the RX 460
Rebooted computer - no boot screen. Desktop shows up shortly afterwards.
About this Mac shows: AMD R9 xxx 4096 MB
System Report > PCI shows: "There was an error while gathering PCI device information" which is unsettling.
When the computer boots, I can see the 460's fans spin up, but they come to a halt when the computer reaches the desktop. If I spin up the fans manually with iStat they still do not spin up.
Any advice?
EDIT: Quick revision - I just opened one of the few games I have, Civ VI, and let it run a benchmark, which seemed to do decent - more importantly, I cracked the case open and took a look at the fans, and they spun up in the game, so that's reassuring.
Still wish I could have the PCI information, since I use it for my ACHI blade as well - is this a cosmetic issue which is fixed if I change the kext information, or does that strictly impact the computer displaying RX 460 instead of R9?