What OS are you using?
I had a similar situation recently with a failed HDD (not Fusion Drive) on a late 2013 iMac that my daughter uses.
Found this iMac for a really good deal on eBay, it was a maxed out BTO, with the exception of the internal drive listed as "for parts", but the symptoms listed from the seller made me think that the problem was most likely a failed internal drive, so I gambled on it, and it ended up being correct.
Even if it was something else, I planned on using the Mac for parts, so I guess it wasn't that risky.
Besides booting a little slower than usual, once booted, the iMac ran great on an external SSD with High Sierra and Mojave. I kept it on Mojave.
Originally planned on replacing the HDD with a SSD, I have done this many times before on an identical Mac and others, but I decided not to due to how well it ran with the external drive.
So, the similar problem....
I got my daughter an identical Late 2013 iMac to keep at her mother's house, with the exception of newer one having a split Fusion Drive.
To keep having me from dealing with potential compatibility issues in the future with a game she plays (WoW), I installed Catalina instead of Mojave.
She brought the wrong computer to her mothers house by accident, and took the one that I purchased earlier in the year. It shouldn't have been a big deal, just plug the external that I set up for her with Catalina, and it should work, no problem.
Wrong....
It took a very long time to boot, ran great for about 20 minutes, then kernel panicked into a restart, just to happen over and over again.
The older iMac ran great with Mojave when the computer was at my home. The newer iMac ran great with the new external with Catalina. Just the older one would kernel panic every 20 minutes with the Catalina external drive.
After some troubleshooting, testing RAM, different OS versions, and Apple useless HW Diagnostic Test, I determined that the failed HDD in the older iMac was clashing with Catalina in a way that didn't happen on older OS versions.
Swapped the failed internal HDD for a SATA SSD in about 15 minutes, and it ran great under Catalina.
Taking the computer apart to remove the SSD physically is more work than it's worth costwise
If you already opened it once to replace the HDD for a SSD, there isn't really that much more work to get the logic board out, imo.