I have the same problem - right now I have the standard Catalina operation system on the iMac's fusion drive that came installed on the machine. And then an external Samsung T5 ssd where I also installed a copy. Would it help if I wipe the fusion drive so that drive is just an internal storage place (which is what I planned) or will the boot up process still be slow because the system is looking if there's anything else besides the ssd that has a boot record?