Sometimes the Apple System Profiler will not report the VRAM,size correctly, you need to use a deeper tool to test the VRAM.
Install Developer Tools and use OpenGL Driver Monitor to monitor the Free VRAM. In a system with Quartz Extreme enabled you can just COMMAND+N a bunch of Safari windows and watch to see how much VRAM is free and in use.
On the AGP 6200 that you have that reports 512MB under OS 9, boot into Open Firmware and get the .properties of the NDVA,Parent@10 to see what the reported VRAM,size is.
If it's 0X2000000 or 0X0100000.
If Open Firmware is reporting the VRAM,size as 512MB( 0x2000000 ) then the issue is likely with OS X drivers, they may read the NVSTRAP and that value needs to match correct for what the driver expect.
There is a reason I keep posting info on the Soft Straps( NVSTRAP ) it's because I already know they have to be edited to get 512MB to work correct under OS X and report the correct amount of VRAM to the ASP( purely cosmetic ).
The info I posted before from nouveau and the info I quoted from Arti should be enough for us to figure out what bits to set for 512MB of VRAM in the strap to get it to work with a Macintosh computer. Forget what makes it work with Linux or Windows.
With PCI Passthrough I tested the PCI 512MB card I have in Emu with SLOF and the FCODE ROM I created with the ROM Maker. I got garbled video out of the card in Open Firmware( SLOF ), however once the nouveau drivers loaded in Debian 11 PPC(LE) the Console loaded with proper video output on the card, tho nouveau reported 1024MB of VRAM and 1024MB of RAM used as the shadow memory( or whatever it's called ) So we know that even the nouveau drivers don't properly report the VRAM,size for the card.
I could not get Xorg up and running in this situation, I could login in the text Console on the card just fine, proper video output on the card, but I get garbled video form Xorg. I'm not real sure the issue here, if I just have an improperly configure Xserver or the are Endianness issues, or it's an issue with PCI Passthrough.
PCI Passthrough needs some special configuration to work with VGA cards and other than me, not a lot of people have ever tested it with older cards, so no one ever fixed the bugs.
These old cards, and PCI cards are mostly untested with PCI Passthrough.
But PCI Passthough is a great way to port HW like these old PCI cards because it sits between the Software and the Hardware and you can dump the register state in realtime while the drivers load and configure the system right to the tty on the host. A "poorman's PCI Analyzer".
The future of PowerPC is emulators, all these old Mac's are going to die and it's going to get harder and harder to find parts and repair them, and the more and more it is going to cost in time and money( and time is money ).
Even PCI Passthrough is only a stop gap. Blanton Zoltan has already made Rage128 emulation work with Qemu PPC, and with a little help from me we got that to work with Mac OS 9. Blanton's target OS the morphOS so he got that working for VGA and even some 2D acceleration. We got basic 'NDRV' support working for OS 9, so you can change the screen resolution and bit depth. It works just as well as the QEMU,VGA device does on PPC. You can even get 32MB of VRAM working on the emulated Rage128 and the ATI Control Panel with properly report all the aspects of the card, including all the VRAM usage buffer.
@joevt work with emulation of the GF6200 in DingusPPC lays the groundwork for register level emulation of GeForce cards.
DosBox and PCemu have register level emulation of the Voodoo2 3D accelerator. It even works in the macOS, but there is a leaked white paper for the Voodoo2 that details all the registers, we don't have this for Rage/Radeon/GeForce cards, so we need to reverse engineer them with the help of the open source Linux drivers and PCI Passthough.
Already my 2022 M2 MPB emulates a PowerMac faster than any real hardware ever built by Apple. With a few drawbacks, the CPU's integer performance is around that of a 1.5Ghz G4, disk IO is way faster than any G4 can achieve other than maybe 64bit SCSI or SATA cards.
Sadly we are kind of limited right now with PCI bus emulation for PCI Passthrough, but that can be fixed.
So we take what we can learn from hacking about in this thread and we build upon it. Sadly Apple didn't leave us in path forward with the macOS on PPC. Beta versions of 10.6 is as far as we can go, but we can still run any software that will run on a real G4 made by Apple in emulation for when the day comes that the hardware is all gone or too expensive to maintain.
The great news about that is 5 years form now G4 emulation will likely be faster than any real G5 ever made.