Quick update: I installed 15.1 developer beta 7, and the issue happens the moment the AMD GPU engages (I have a iStat Pro set up to tell me which GPU is active).
I'm extremely disappointed with apples, software side of things. Are used to get 10 years out of a laptop, but considering that I'm seeing some people report this issue in Ventura (maybe I'm misremembering?), that's essentially only two years of years from when this device was released, before the OS became incredibly buggy on it. I really love Apple Silicon, but I still wanted to get a few more years out of this machine, and more importantly, I have a need for running Intel virtual machines.
It's not even like this is the only bug this has, I have to unplug it to use touch ID in Sonoma and Sequoia.