That is normal. When the machine is in idle, macOS does all the housekeeping and background tasks. By default background tasks are scheduled with reduced priority which makes them not run at all when you use your machine. (there's a command that makes them run at regular priority. For example to speed up time machine backups).
This would make background tasks run without throttle.
Bash:
sudo sysctl debug.lowpri_throttle_enabled=0
A reboot or this reverts it back to default
Bash:
sudo sysctl debug.lowpri_throttle_enabled=1
In idle, macOS also preemptively swaps ram to disk or compresses memory if not disabled and "if necessary" (Apple's definition of "necessary" is a bit different from mine here, but ok).