POSSIBLE RESOLUTION 2023/01/17
Was experiencing this same problem of a memory leak and GB of memory use and swap files. Monitored the console for ARDAgent and noticed it kept clearing a queue. Also was having trouble keeping the menu bar item for remote agent visible.
Figured it was something to do with that, tried disabling the menu bar item and fixing permissions but still no luck, next tried removing antivirus and that didn't help.
I noticed that I kept Caffeine in my menu bar in turned off mode from my startup items, I quit the Caffeine app and the noise stopped in the console app. Disabled it from the startup items and rebooted and the console is completely quiet for ARDAgent. Reinstalled antivirus and still quiet.
Think this solved the problem, will check in tomorrow to see if there's any more memory leaks. If resolved maybe someone needs to alert the developer of this issue in Caffeine.
UPDATE 2023/01/18
One day later I checked in on ARDAgent and it was up to 7GB of ram, so it didn't solve the problem but slowed it down some.
Maybe related to other menu items, I'm running Menu Meters and SMC Fan Control and Avast besides Apple's items.
If I figure it out I'll let you know.