OP wrote:
"Turning it off and on, or keeping it off? I only have a BT wireless mouse and keyboard so that's not really an option."
Well then...
Perhaps it's time to consider a different keyboard and mouse setup.
I would suggest a WIRED USB keyboard.
If you can get one with a USBa port built in, that would be ideal.
But it's not absolutely necessary.
For a mouse, I'd suggest a Logitech wireless mouse (something that comes with Logitech's "unifying receiver". This IS NOT bluetooth, but rather a dedicated radio channel.
IF your iMac has USBa ports, plug the unifying receiver into one of the "A ports".
If you have an m1 iMac with only USBc ports, then you need a way to connect the unifying receiver. It's a little more complicated, but doable.