My thoughts to your concerns are... YES... daily "unplugging and re-plugging" WILL wear out the ports sooner than they would otherwise wear out.
If you've ever delved into the MacBook area here at macrumors, you might have seen posts by some users who complain that their HDMI port (on their MacBook Pros) has "gone loose" on them, quite early in fact.
I have a possible solution for you:
Get a used/new-old-stock Caldigit TS3Plus dock.
Can be found on ebay for under $100.
This is a thunderbolt 3 dock, NOT a tbolt4 or tbolt5 dock.
But... for connections that don't need tbolt4, it should work fine.
Make all your peripheral connections to the dock.
Use a single tbolt3 cable to connect to the Mini.
That's still "one connection" to be made/broken, but only one.
The rest should take care of themselves in the dock.
Hmmm... you'd need a second dock "for work"...
But, it is, what it is.
No Macs at work?