Just called the support. They asked me to send it back, I'll get a replacement or a repair.Another use suggested turning special audio to fixed and that may have solved it for me but I’ll have to pay more attention if it pops up again.
Hopefully this is addressed soon among other little bugs with the AirPods. I still also get random disconnects for example
After thinking about the problem it seems to me, that the AirPods loose the connection between each other and then act as two single mono speakers which causes this weird spatial sound. The few milliseconds delay between the two lead to this echo-effect. This might also cause connection errors, when two concurring but not communicating AirPods with the same device name are connected to one iPhone.