There are now much better MXM3 cards with Metal support for the iMac 9,1 > but only the older
MXM2 cards will fit
the iMac 7,1 and 8,1 MXM2 slot, and no MXM2 cards support Metal. Some people tried these Quadro FX cards years ago,
I think there was a thread on the old Netkas forum, gone now but luckily I managed to find some of the iMac rom files.
The FX 770 probably didn't work because the LCD configuration is wrong, just showing a black screen or weird colors?
Also the video bios chip on the FX 770M card is only 64K, which is much too small to hold a full Mac EFI video bios.
You can try to flash this GT120 rom (same Nvidia G96 chip) but this rom file is missing the EFI part, so no boot screen,
and it might just totally break the FX 770M card if something does not match (memory types etc) - so I wouldn't risk it!
https://www.techpowerup.com/vgabios/167063/167063
But... There are two Quadro FX cards that can be flashed to work in these iMacs, both need the larger 8800GS heatsink.
You can flash the
Quadro FX 2700M card with this patched GT130 rom, but these cards also only have a 64K rom chip,
so you would have to solder on a larger 256K chip. These cards were only used in a few laptops so are hard to find now.
Its
G94 chip is not very fast but at least they will not overheat and fail like the faulty G92 on the original 8800GS card.
Probably the easier card to use is the
Quadro FX 3600M, it has the same Nvidia G92 chip as the Apple 8800GS card,
but a newer revision that should not fail as quickly. They come with a 128KB bios chip that is just large enough to hold
the full 8800GS video bios with the EFI part, so they show boot screens and work in MacOS just like a real Apple card!
There were lots of FX 3600M cards available from Laptops and HP Proliant servers (fitted to Mezzanine MXM adapters)
Search HP part codes 580138-B21 / 490567-001 / 492189-001. Flash them with NVflash in windows, a linux boot disk,
or using a CH341A with a clip adapter on the EEPROM chip, see the iMac video card upgrade thread for flashing details.
The iMac vbios rom files are attached here, I hope they help save a few more old iMacs from the trash!