Same. Not one panic since disconnecting my Time Machine drive.
How is the drive connected, what kind of drive is it and what type of Mac? I never had any kernel panic and use many external drives.
My TimeMachine drive is at the moment a very small 4TB SSD literally (because it's so light) hanging directly on the back of the iMac.
I also used two different 8TB HDDs with external power on a TB4 Dock until recently for Time Machine. Maximum was 3 drives at once in TM.
The HDDs are now used for Carbon Copy Cloner because it works with HFS and I somehow don't like APFS, especially for HDDs.
Disks get very often corrupted, especially if you work with partitions and then they have to be completely reformatted. Also SSDs. They work, but I can't delete a partition and Disk Utility says it's corrupted.
There is also a Thunderbolt SSD that is connected to the dock in Carbon Copy Cloner that should be bootable. But it doesn't really work to boot (gets stuck at the beginning of the progress bar) and I can't use it as my main drive. It's even faster than the internal 1TB. Same problem for a "bootable" clone I created with SuperDuper, it tries but can't boot.
All in all I have 4 SSDs, 2x the same mini 4TB (USB 3.1 or 3.2 Gen2 with 10Gbit/s) without external power, 2x SSD (2+4TB) TB3 or 4 with 40 Gbit/s and two different external powered very big with included USB-Hub 8TB USB 3.0 HDDs (5GBit/s connection). Two of those drives have more than one partition. And a small 4TB USB 3.0 HDD without external power is sometime connected too. Some are connected to USB-A and some to USB-C.
I can connected them all together to my iMac M3 without any problems. Only one thing is a little annoying often a disk can't be ejected because it's still in use somehow and I don't know why. I have to click on force eject then. Or just power down the dock like I do every evening. At least self ejections don't happen very rarely now. I think that mostly happened with directly connected USB-powered older SSDs.
Normally I disconnected all drives directly after the daily backup especially the "loud" 8TB HDDs except the TimeMachine drive and the TB4 SSD for fast manual backups. Sometimes I forget it and they all are there the whole day. I have some old drives and sticks that I can connected in addition without any problems.
A second display and a SoundBlaster is also connected, the Display to Thunderbolt and the SoundBlaster to USB. Both on the Dock.
Also Two USB-Hubs, that don't need any extra power for Mouse, two Keyboards, Yubikeys etc.
I tried even tired to connect a TB3 Dock on my TB4 Dock and a TB2 Dock and USB-C Dock on one of the Docks or directly on the iMac. That really worked. At least all ports where listed in "System Information".
I also have a second TB4 dock what I wanted to use for my MBP. It needs two ports and I can connect the second TB4 dock on that one too.
I really regret buying all those Docks. Especially the TB4 ones that costed almost as much as a Mac Mini. TB4 cases for SSDs are also very expensive. I just saw yesterday Apple ist selling a 1m TB5 cable for €79. Before I thought TB3 and 4 cables are expensive, but I paid maybe a little more than 20 Euro for a 2m TB4 cable recently.
Still hoping for an update because of some annoying other problems. At least no kernel panics, only some apps are crashing sometimes and there is this AppStore issue. I am happy that most of my apps are not from the AppStore.
AdGuard is also doing strange things. But that's was already before this beta. Sometimes it just turns off and says it can't be used and I have to turn it on again several times.