The screen going blank means AllRez is testing an attribute or DDC command that the display doesn't like. I'll need to test with different GPUs. You can try the old version (from the git releases) to see if it's a problem only with the new version.
All the files have the same EDID (13 occurrences) which is expected.
AGDCDiagnose_resolution.txt is similar to AGDCDiagnose_a.txt - both connect at HBR3 x4.
AGDCDiagnose_usb.txt shows a connection at HBR3 x2.
The allrez files show the same link rate/link width differences.
For some strange reason, allrez-usb.txt says the startup-timing is 10bpc while allrez-resolution.txt says 8bpc. I guess this doesn't matter.
AllRez was able to read the DPCD of both the MST hub and the sink device.
The sink device has support for DSC. I don't know how this works if the MST hub doesn't also have support for DSC? The DisplayPort 1.4 spec is not available for download, and the Linux code is hard to follow (I suspect the DisplayPort 1.4 spec is also hard to follow, but might be more informative). Can an MST hub that doesn't understand DSC just pass a DSC stream to a sink device that does support DSC?
I don't know what kind of MST device this is or exactly how MST works - for some reason, the sink device has the same link width as the MST hub's upstream connection. Maybe it's a really simple MST hub that doesn't know how to convert link width from 2 to 4 - but this doesn't matter as long as the sink device supports the same link rate (meaning that the connection between the sink device and the MST hub is not a bottleneck - in this case, the only bottleneck is the connection between the MST hub and the GPU).
The current mode in both cases is RGB 10bpc framebuffer and 8bpc output, 5K scaled down to 4K for 2560x1440 HiDPI mode, not HDR.
If the display has 4 lanes of HBR3 when prioritizing resolution, why is macOS creating 4K60 display modes only at 8bpc? Did you disconnect the display when you switched from prioritizing USB?
The following script calculates the Gbps from the timing info (only works for modern macOS GPU drivers that include bpc in the timing info) and sorts them by Gbps. The code assumes you have only one display connected (otherwise I would have to add code to the script to separate the modes of each display).
Code:
thefile="allrez-resolution.txt"
perl -nE '
if (/{ DetailedTimingInformation = { V2 .* (\d+.\d+)MHz.*active:(\d+x\d+).*encodings\((\w+),\) bpc\((\d+),\) colorimetry\((\w+),\) dynamicrange\((\w+),\) dsc\(\d+x\d+ (\d+)bpp\)/) {
print "Gbps:" . $1 * ($7 ? $7 : $4 * 3) / 1000.0 . " MHz:" . $1 . " active:" . $2 . " encoding:" . $3 . " bpc:" . $4 . " colorimetry" . $5 . " dynamicrange:" . $6 . " dsc:" . $7 . "\n";
}
' "$thefile" | sort -V
You see in the results that macOS doesn't create modes greater than 12.798 Gbps.
HBR3 x2 is 12.96 Gbps
HBR2 x4 is 17.28 Gbps
HBR3 x4 is 25.92 Gbps
So in your case, you are being limited, even though you have "prioritize for resolution" selected. Try disconnecting and reconnecting the monitor to see if you can get 4K60 10bpc RGB instead of just 8bpc.
