I always found that to be a terrible experience, especially with any sort of video playback. Even over a direct Thunderbolt connection it was very laggy. This was an older generation of Mac mini though (2012). It was usable enough for some basic tasks but anything with full-screen graphics like video playback was out of the question due to the lag.I just parked my new Mac mini under the side of the iMac screen, turned on Screen Sharing on the iMac, and the iMac became the display for the Mac mini. I use two FullScreens on the iMac and flick the mouse between the iMac machine and the Mac mini machine. (Note: borrowed an older 4k monitor to set the Mac mini's default screen.)
How is the lag for you?