OP: I'd play process of elimination to figure out exactly what is the problem. Too many variables to know from what is described so far.
Can you see this happen with a single video? Will it consistently stutter in the same spot if you play it over and over? If so, try playing it with other players such as the native QuickTime app. If it consistently stutters, you'll rule out a possible Kodi & VLC apps issue. The "works with Linux laptop" concept is changing the variables of the apps too.
Can you temporarily connect your Mini to your router with an ethernet cable just to test wifi vs. ethernet. Again, play a video you know will stutter and see if it stutters on ethernet too. If it doesn't, you dig in deeper on wifi catalysts.
Yes, try a different cable to rule out/in a faulty cable.
Yes, try a different drive to rule out/in the drive.
If there a hub between drive and Mac? If so, try a direct connection to rule out/in hub.
Could hard drive be going to sleep and needing time to wake for the next block of video to be streamed? The test by copying one problematic video to the internal SSD would help explore this idea. If it turns out to be this, the solution is not letting the drive sleep.
Is it consistent with many files or only 1 or a few? Anything odd about those files? Is the format Apple friendly?
If the stutter is always in the same spot, perhaps it's just a bad encode.
Consider identifying a few files doing this by name in Kodi or VLC and others with the same setup could potentially try playing the very same files to either confirm the problem beyond your hardware or refute it (so you know it is in your setup alone).
The path to a solution only really becomes possible when you pin down the problem as narrowly as possible. As is, it could be ports, macOS, Mac mini M4, file, drive, cable, enclosure, format, wifi, etc. Find yourself one file that regularly does what you see and start working through process of elimination. For example, if that file plays fine from the internal drive through Quicktime, change ONE variable to see if it will stutter. Then change another variable.