I guess you want to say HD5870, there is no Mac Pro shipped with 5850.
The monitor size won’t affect the GPU performance. Resolution will, if in 3D environment etc. But for video play back, that’s almost nothing to do with the GPU performance in MacOS. It’s hard to believe that the 5870 can’t play a normal full HD video smoothly. But if you play something like 4k H265, then it’s the CPU’s job to decode, no matter which GPU you use, MacOS won’t use it’s hardware decode ability. If your video playback is choppy now, that may be still choppy even you upgrade to a TitanX.
IMO, the only bet now is Vega. Since the coming iMac Pro will ship with the Xeon that I believe has no iGPU (no Intel QuickSync), and I don’t think Apple can afford the iMac Pro looks stupidly slow in the video encoding test etc. Therefore, they may activate Vega’s dGPU hardware encode / decode ability. For Apple’s style, there is a high chance that we need a Vega has identical Device ID as the iMac Pro’s Vega (or even need some hack) in order to benefit from it. However, it’s still just my guess. Since Apple may only care how the iMac looks but not how it perform, there may be no hardware encode / decode ability at all for the iMac Pro.
Anyway, for video playback, try IINA. In my experience, that’s better than VLC in MacOS.
For GPU commendation, you really need to tell us do you require boot screen. If that’s just preferred, and you have a stock Apple GPU on hand now. The recommendation can be very different.