Updated from Mojave to Big Sur a couple of weeks ago, figured the coast was clear (so to speak). But no, I've run into this bug as well (Magic Keyboard and Magic Mouse on a 2017 27" iMac). As mappyuk noted, seems to happen after waking the iMac. Another weird side effect is that the diacritical mark popup often appears when trying to type in this situation. I've reported this collection of problems to Apple as a bug.
I just resolved the problem by consecutively:
1. Resetting the SMC. (disconnect the power cable from the back of the iMac, wait 15 sec., connect again and restart.
2. Zapped the NV PRAM (cmd-opt-P-R and I let it chime 3 times). After restart:
3. opt-shift >Bluetooth menubar> Reset Bluetooth module. (still nothing)
4. opt-shift >Bluetooth menubar> Factory reset all connected Apple devices. (this caused the trackpad to be unresponsive, so I had to hook up a wired mouse and:
5. Deleted the Bluetooth prefs in the User Library Preferences.
Then I restarted (still using the wired mouse) and the Bluetooth setup window appeared.
The Trackpad showed up and asked to connect.
Trackpad connected, and no lag anymore in the keyboard or Trackpad.