Here someone was unable to get 4K60 from a 2013 Mac Pro, presumably using DisplayPort (no mention of HDMI/USB-C).
This post shows 4K 60Hz 10bpc HBR2 :
#275 on the U3223QE but this connection is after the MST Hub which might be converting from HBR3. Need AGDCDiagnose to know what's going on.
This post
#298 says there's a switch to change USB-C input from 2 lane with USB 3 to 4 lane with USB 2.0
This post
#421 explains the switch for changing the upstream USB connection so you don't need to be limited to USB 2.0 (in some situations).
If a display requires 2 lanes of HBR3, then I suppose an external DisplayPort 1.4 MST hub could be used to convert 4 lanes of HBR2 to 2 lanes of HBR3. The HP Thunderbolt Dock G2 is the only DisplayPort 1.4 MST Hub that I know that has a USB-C DisplayPort Alt Mode output in case you want a solution that includes USB 3.x. You should be able to connect it to a 2013 Mac Pro with a Apple Thunderbolt 3 to Thunderbolt 2 Adapter. But you might need to put a Thunderbolt 3 device between the Thunderbolt 2 Mac and the HP Thunderbolt Dock G2 (I could not get the Apple adapter to work with the HP dock directly so I connected the HP dock to a OWC Thunderbolt 3 dock).
I did some tests:
- I connected a HP Thunderbolt Dock G2 with reduced Thunderbolt 20 Gbps connection to my Mac mini 2018 to match more closely the Mac Pro 2013. This has a 4 lane HBR2 connection to the iGPU (UHD 630).
- I connected a CalDigit SOHO to the USB-C DisplayPort Alt Mode port of the HP dock. It's a USB-C dock so it has 2 lanes of HBR3. This is analogous to the Dell's MST Hub.
I tried several 2560x1440 modes with refresh rates from 60Hz to 120Hz (CVT-RB). The max I could get working is 104Hz which has a pixel clock of 428MHz. 105Hz has a pixel clock of 432.31MHz. I think I'm running into the pixel clock limit for HBR3 x2 @10bpc which is 432MHz. macOS thinks the max pixel clock is 940MHz (maybe this corresponds to HBR3 x2 8bpc 4:2:0 or HBR3 x4 8bpc RGB which is 1080MHz - it's not taking into account the MST chain?). The problem is that the modes above 432MHz do not switch from 10bpc to 8bpc until between 572.56 and 577.31MHz (except modes at 594MHz which remain at 10bpc for some unknown reason - the 594 MHz modes don't indicate chroma sub sampling...). This switch to 8bpc corresponds to the 8bpc limit for HBR2 x4 (576MHz). The problem with that is the MST chain has a HBR3 x2 bottleneck (75% the bandwidth of HBR2 x4).
We need a OpenCore/Lilu/WhateverGreen patch to fix this incorrect bandwidth calculation or to add a method to change bpc, RGB/YCbCr, DSC bpp, etc. Of course, these tests are done using the more modern Intel GPU drivers. The AMD D500 or D700 drivers may have different behavior.
I want to add a method to my AllRez command to output DPCD for all MST hubs in a chain but it might require OpenCore/Lilu/WhateverGreen to get access to the DPCD functions of the driver because the user I2C APIs don't have a method to communicate with DisplayPort devices that are not directly connected to the GPU.