There is no GPU hardware decoding on your cMP.
So, all video decode has to be handled by CPU via software method. Which means, different software use different way to decode the video.
In your case, VLC has a lower demanding / higher efficiency method to decode the video. Therefore, it can play the same video smoothly.
The older QuickTime has the oldest (most ineffective) way to decode the video, and the cMP's CPU isn't fast enough, therefore, repeatedly freezes.
The newer QuickTime has newer (better, but still not good enough) way to decode the video. Therefore, less shuttering, but still not smooth.
The bottleneck isn't the GPU, but the CPU. This is why I spent years to figure out how to make GPU hardware decoding / encoding works on the cMP in macOS.
[For existing OpenCore users, you can always download my latest OpenCore package at post #1314 for update] Full HWAccel achieved now! Everything is covered in the OpenCore thread. The step by step tutorial to enable AMD hardware acceleration is now at here, but I strongly recommend you go...
forums.macrumors.com
This can move the workload from CPU to GPU, and remove the bottleneck.
N.B. Only some H264 and HEVC videos can be decoded by GPU on the cMP. No matter how we upgrade the graphic card or macOS, this is still the best we can get. However, in Windows, we may use the same GPU to decode VP9 or AV1 etc, depends on which GPU you are using.
If you are still running High Sierra or below, upgrade the GPU won't help.