You have to show the timing in SwitchResX to prove it's running at 5K. The signal being sent to the XDR is probably only 4K since Thunderbolt 2 can only transmit one 4K signal. In SwitchResX, double click the current resolution in the Current Resolutions tab to view the timing. It will show Pixel Clock, Active, Scaled To, Scan Rate for the current resolution.
The MacPro6,1 only support DisplayPort 1.2 but HDR require DisplayPort 1.4. You can get 10 bpc color with DisplayPort 1.2 but it won't be HDR.
The following commands will show the connection information for the display (DisplayPort version, link speed, number of lanes, Display Stream Compression capability/status/enabled, supported timings, etc.
Code:
/System/Library/Extensions/AppleGraphicsControl.kext/Contents/MacOS/AGDCDiagnose -a > AGDCDiagnose.txt 2>&1
/System/Library/Extensions/AppleGraphicsControl.kext/Contents/MacOS/DisplayDiagnose -a > DisplayDiagnose.txt 2>&1
The XDR will only work at 4K with the D500. With one XDR, you can run 5 ACDs. To run the XDR at 5K or 6K, you need to connect a Blackmagic EGPU. Apple doesn't support eGPUs with Thunderbolt 2 Macs (for no good reason) so you'll need to use a workaround (visit eGPU.io website, and look at existing Builds).
Without an eGPU, you might be able to get 5K by connecting your MacPro6,1 to a GC-TITAN RIDGE with two mini DisplayPort cables, then connect the XDR to the GC-TITAN RIDGE. You will be missing some features though.