I get the same on my 2017 27" iMac and I think it started happening after the Safari 14 update. I'm also using a magic keyboard (with keypad) and trackpad. It's intermittent and happens a few times a day.
Could this be the cause if I hardly ever use Safari? I primarily use Firefox but I experience the lag no matter which programs I have open.
I've had problems in the past with certain USB devices interfering with my mouse. I think it was an external SSD. What if anything do you have on your USB ports?
Currently, I have an external SSD, audio interface, thumb drive, Apple SuperDrive, a wireless charger, and an external webcam plugged in. These were all plugged in and working fine for (at least) weeks before I started experiencing problems.
As an added note, it is definitely more than the mouse and trackpad that are affected. It seems like the whole system slows down for 10-15 seconds at a time, and I can sometimes notice it when the input from my keyboard slows down.