Just because your iClloud devices show the Airpods, I wonder if they really are "paired" in the traditional Bluetooth sense. Or is Apple tweaking the user interface like when you turn off Bluetooth in the iOS Control Center, it's not really completely "off". So maybe when you switch Airpods between devices, that is when the device actually "pairs" with the Airpods, and it has to do this every time you switch devices which would explain the apparent latency when switching devices?
For comparison, I had a Jabra Sport Bluetooth headset which could be paired with 2 devices at the same time. Once the Jabra was paired, switching between devices felt much faster compared to the Airpods.