There are no USB4 or TB3 KVMs. It would need to be able to switch 20 Gbps signals.
There did exist Thunderbolt 2 KVMs but I don't know if any are available. They won't work for 6K from BlackMagic eGPU or other GPUs that don't support DSC. the Sonnet eGPU Breakaway Puck RX 5500XT/5700 support DSC so they could work with a Thunderbolt 2 connection but in that case you might as well use a DisplayPort KVM (except Thunderbolt 2 KVM would allow USB 3.0 speed from the XDR USB ports).
https://www.kvm-switches-online.com/thunderbolt-kvm-switch.html
https://www.iogear.com/product/GTD732/
I think it would be possible for Apple or someone to create a kind of Thunderbolt Target Display Mode that would allow you to switch a Thunderbolt connected display between multiple Macs that are connected together using Thunderbolt. Basically the host Mac (the one that controls the XDR) would allow a cross-domain tunnelled DisplayPort path from another Mac to the XDR (two DisplayPort paths in the case of Macs that don't support DSC which might be tricky since those paths would use up 39 Gbps - currently Apple doesn't allow such paths except when there is nothing between the Mac and the XDR).
I don't think Universal Control is going to be much better than screen sharing for using the XDR with other Macs. Therefore the only thing you can do for now is keep disconnecting and reconnecting the Thunderbolt cable.