MP 7,1 MPX Radeon Pro VII Duo runs two XDRs fine … until I install a second graphics card? WTF?

[elliottcable]

So, this is super, super weird.

I've got a 2019 Mac Pro spec'd out with the Radeon Pro Vega II Duo (MPX module), and two Pro Display XDRs attached to the VII Duo via Thunderbolt 3. As officially supported, of course:

Prior to purchasing this Mac Pro, I was using an eGPU (Sonnet Breakaway Box) with an AMD Radeon Pro WX 9100 — again, an officially supported configuration!

I have non-Apple displays I need to drive, as well as the Pro Display XDRs; when purchasing the Mac Pro, my plan was to simply move my WX 9100 (an expensive piece of kit on its own!) into the Mac Pro (isn't that the whole point of the return to the standardized, expandable chassis? ), and continue using it to push my non-Apple pixels.

Well, bad luck:


Whenever a second graphics card is installed in my Mac Pro — not even connected to any displays! — the Vega II Duo will only drive a single Pro Display XDR at full resolution.


If I connect the second XDR to my VII Duo, it, oddly, happily reports that it's running at full 6016×3384; but text is visibly fuzzy; it very definitely looks like it's being downscaled (poorly!) to 5k. It's especially obvious that something is wrong, as it's sitting right beside another XDR that's actually functioning at full, sharp 6k.

Further oddness: With the WX 9100 installed, I have to boot up with only a single XDR connected to the VII to get either of them to function at 6k; if I boot the machine with the WX 9100 installed *and* two XDRs connected, both XDRs are fuzzy-6k-scaled-up-from-5k.

And the final odd component here? Everything works swimmingly if I connected the WX 9100 to the Mac Pro in my eGPU chassis. That's right, there's 6 unpopulated PCIe slots in this picture, and yet I'm using an eGPU chassis, because that's the only way to get all my displays working:

At this point, I feel I may be the only person, including engineers at Apple, who's ever plugged an eGPU into a Mac Pro. Ugh. >,>

So, tl;dr:

• Is anybody else currently running MPX and non-MPX graphics cards, simultaneously, in a Mac Pro?
• If so, are you experiencing any problems with display resolutions?
• Has anyone seen a Pro Display XDR reporting a resolution of 6016×3384, while clearly not being there? /=

 

[krakman]

###WILD GUESS###

have you used the PCIE expansion slot utility app to adjust bandwidth between the cards when installed inside the mac PRO?

 

[bcomer]

Slots one and three are connected directly to the CPU and should be used for the two cards.
All the other slots go through the PCI switch. Try putting the 9100 into slot three providing the MPX module is plugged into slot one.

 

[Ludacrisvp]

And here i'm just curious to know why you chose the rackmount version of the 7,1.

 

[cobra521]

Another WAG: Have you tried swapping cables around to see if one of them might be a cause?

 

[awkward_eagle]

###WILD GUESS###

have you used the PCIE expansion slot utility app to adjust bandwidth between the cards when installed inside the mac PRO?

Sounds like the eGPU is sucking up some of the TB3 bandwidth needed for the monitor.

 

[nicho]

Sounds like the eGPU is sucking up some of the TB3 bandwidth needed for the monitor.

Eh it sounds about as far from that being the problem as it possibly could, to be honest.

 

[awkward_eagle]

Eh it sounds about as far from that being the problem as it possibly could, to be honest.

Thanks for the comment. Really insightful.

To the OP:
Using the slot utility to reroute the bandwidth should let you know either way. Since xdr modules route the video signal over the tb bus they could very well be saturated. Using egpu box probably just moved which bus was being used.

 

[Snow Tiger]

I have non-Apple displays I need to drive, as well as the Pro Display XDRs; when purchasing the Mac Pro, my plan was to simply move my WX 9100 (an expensive piece of kit on its own!) into the Mac Pro

Overall , your issue does sound like the Thunderbolt PCIe lanes are being redirected somewhere else .

Is there some reason why you needed to install a WX 9100 ? Drivers , ECC , 24 hour tech support ?

An AMD Vega Frontier Edition is basically the same thing as a WX 9100 at a substantial savings .

 

[tommy chen]

And here i'm just curious to know why you chose the rackmount version of the 7,1.

you come to the inside without having to stake out everything
and I find it more beautiful - would also be my choice

 

[elliottcable]

And here i'm just curious to know why you chose the rackmount version of the 7,1.

Variety of reasons — most of them being that I've got very specific space constraints in my laboratory, and just didn't have room outside of the exactly where my rack is going in. If I'd gotten a tower, it'd be … sitting on top of the rack.

Slots one and three are connected directly to the CPU and should be used for the two cards.
All the other slots go through the PCI switch. Try putting the 9100 into slot three providing the MPX module is plugged into slot one.

I did, actually, have it installed in that slot. )=

###WILD GUESS###

have you used the PCIE expansion slot utility app to adjust bandwidth between the cards when installed inside the mac PRO?

None of the cards are adjustable, oddly. The WX 9100 is listed as an "unknown card", and dosen't have re-assignment options next to it.

Another WAG: Have you tried swapping cables around to see if one of them might be a cause?

I haven't! The two I was using are the first-party ones from Apple; but I do have another 2m 40Gbps active one from Belkin … I'll try that out when I've got it reassembled into the broken-state for my phone call with Apple's advanced support this afternoon. (Remember, though, that the cables/displays all work fine when the WX 9100 is uninstalled … )=

Sounds like the eGPU is sucking up some of the TB3 bandwidth needed for the monitor.

Er. When the eGPU is being used is the only time everything works. I think you mis-read my original post, sorry.

Is there some reason why you needed to install a WX 9100 ? Drivers , ECC , 24 hour tech support ?

An AMD Vega Frontier Edition is basically the same thing as a WX 9100 at a substantial savings .

Right now? Simply because I already had it. Previously? Because:

1. It was the only card in existence with enough ports for what I was doing (2×MST 5ks, requiring two DisplayPort cables each; 2×SST 4ks, requiring two more DisplayPort cables, for a total of 6 DB cables to one card.), and …
2. because it was on sale with the Sonnet Breakaway Box in a package. Served me excellently for years; and continuing to use it costs me a hell of a lot less than trying to sell it and buy another MPX module — the two XDRs completely saturate the VII Duo, but I have another display I need to drive.

 

[awkward_eagle]

Er. When the eGPU is being used is the only time everything works. I think you mis-read my original post, sorry.

You're absolutely right. Should have read it twice. Was thinking thunderbolt bus but typed out egpu instead. My thought was that the mpx modules are routing the signal to the off-gpu thunderbolt ports. Are the XDRs plugged into the mpx tb3 ports or off-card ports?

Either way it sounds like either a load balancing issue or a driver bug. Wouldn't be surprised if adding a second non mpx gpu conflicts with what Apple is doing to combine multiple displayport streams into a single image buffer to be sent over a tb3 cable. Since it's a proprietary Apple thing we may never know unless they publish a recommended config guide. Could also be a Catalina display driver bug which hasn't been the most reliable.

 

[elliottcable]

Holy crap. It makes absolutely no sense, based on the documentation ...

... but moving my second graphics card, the WX 9100, into a switched PCIe slot (I chose Slot 5) actually fixed the problem.

Thing is, Slot 3, where it previously was, is directly connected to the CPU. And absolutely shouldn't be competing with Slot 1 for PCIe bandwidth. What the hell.

Anyway, well, something learned, I suppose: until a firmware update to fix this, if you're using a non-MPX graphics card in the Mac Pro and want Pro XDR displays to work, install it in Slot 5?

 

[straightMacin]

…
[*]because it was on sale with the Sonnet Breakaway Box in a package. Served me excellently for years; and continuing to use it costs me a hell of a lot less than trying to sell it and buy another MPX module — the two XDRs completely saturate the VII Duo, but I have another display I need to drive.
[/LIST]

I find this bit interesting. You can't drive additional displays beyond 2x XDR's with the Duo? That's disheartening for my case of needing to drive 3 with just a single Vega II single. I just ordered a W5700 with hopes that can power the one additional XDR as well as 2x Thunderbolt displays.

 

[s66]

Which TB3 connectors were you using ?
The VEGA II duo has multiple TB3 busses with multiple connectors on each.

the back of the card has 2 TB3 busses with 2 connectors each, the 2 connectors on the same bus as side-by-side

The rest of the displayport signals are directed into the backplane on the computer to go to the other 4 TB3 ports on the system and the IO card.
The bandwidth to do that is provided outside of the normal PCIe connection on the extra connector the MPX modules have.

What you describe looks like the system is setup to not let a single VEGA II (DUO) send 2 displayport feeds onto the MPX connector once it knows both slots for the MPX modules are in use and only allow one displayport feed fro each MPX slot.
If you plug in both displays onto the ports on the back, makign sure not to use twice the same bus, is it working then with a card in the second MPX slot?

 

[nicho]

Thanks for the comment. Really insightful.

You’re welcome.

 

[tsialex]

Holy crap. It makes absolutely no sense, based on the documentation ...

View attachment 897444

... but moving my second graphics card, the WX 9100, into a switched PCIe slot (I chose Slot 5) actually fixed the problem.

View attachment 897445

Thing is, Slot 3, where it previously was, is directly connected to the CPU. And absolutely shouldn't be competing with Slot 1 for PCIe bandwidth. What the hell.

Anyway, well, something learned, I suppose: until a firmware update to fix this, if you're using a non-MPX graphics card in the Mac Pro and want Pro XDR displays to work, install it in Slot 5?

Your problem is not related to lane allocation, or even Thunderbolt, but the way that Apple routes the video and audio signals from the MPX modules to the Thunderbolt ports.

Both MPX slots can route video/audio signals, while the other PCIe slots below the PCIe switch can't. Using the MPX slot, you probably are over allocation the muxes and signal (audio/video/USB) passive switches that do all the routing between the video cards, the top case TB ports and the I/O card. This is complex to understand but if you look at the MPX slots, there are several passive components and the other slots don't.

 

[elliottcable]

Which TB3 connectors were you using ?
The VEGA II duo has multiple TB3 busses with multiple connectors on each.

the back of the card has 2 TB3 busses with 2 connectors each, the 2 connectors on the same bus as side-by-side

The rest of the displayport signals are directed into the backplane on the computer to go to the other 4 TB3 ports on the system and the IO card.
The bandwidth to do that is provided outside of the normal PCIe connection on the extra connector the MPX modules have.

What you describe looks like the system is setup to not let a single VEGA II (DUO) send 2 displayport feeds onto the MPX connector once it knows both slots for the MPX modules are in use and only allow one displayport feed fro each MPX slot.
If you plug in both displays onto the ports on the back, makign sure not to use twice the same bus, is it working then with a card in the second MPX slot?

Yeah, I'd already read the docs on that; all of my tests were with the XDRs connected to ports 1 and 3 (so, bus 0 and bus 1, respectively) of the VII Duo. No dice, until I moved my other card out of the MPX-allocated PCIe slot.

Your problem is not related to lane allocation, or even Thunderbolt, but the way that Apple routes the video and audio signals from the MPX modules to the Thunderbolt ports.

Both MPX slots can route video/audio signals, while the other PCIe slots below the PCIe switch can't. Using the MPX slot, you probably are over allocation the muxes and signal (audio/video/USB) passive switches that do all the routing between the video cards, the top case TB ports and the I/O card. This is complex to understand but if you look at the MPX slots, there are several passive components and the other slots don't.

I can't directly controvert this; but it'd be really, really odd if that were the case: unless something is actively plugged into the second, proprietary MPX connector at the rear of Slot 3, then the bandwith that would be allocated there continues to be allocated to Slot 4 — so why would the DisplayPort routing switch based on presence-in-PCIe-slot instead of presence-in-MPX-slot? That'd be a really poor design.

That said, your idea does track with the symptoms ... idk, I still suspect firmware, and hope an update will fix it, personally.

 

[gabrielefx]

first try to put your Radeon Pro into an 8X pci slot, second, don't use MPX bay to install non Apple gpus. Probably MacOS thinks you have a second Vega and get confused with tb3 pci-lanes assignment. I think you can connect the third monitor with an adapter to the top or back tb3 ports. You don't need a third gpu except you want to use it with DaVinci Resolve Studio or you use OpenCL/Metal 3d rendering programs (Blender).

edit:
I see you fixed the issue.
Also I think the pci lanes scheme is wrong. The MPX back slot is another pci-4X bus used only by the thunderbolt ports. I don't know of these two slots are routed to the Titan Ridge controller and directly to the Xeon cpu, probably they pass through the pci-switch.
Two gpus on the same 16X slot (Vega Pro II Duo) is not a good choice because they share the same 16X bus. It was better to buy two Vega Pro II.

 

[joevt]

Yeah, I'd already read the docs on that; all of my tests were with the XDRs connected to ports 1 and 3 (so, bus 0 and bus 1, respectively) of the VII Duo. No dice, until I moved my other card out of the MPX-allocated PCIe slot.

That's really weird. Any of the following should work for connecting two XDR displays to one Vega II or Vega II Duo:

Vega Port numbers (one XDR display connected to Thunderbolt Bus 1 and another connected to Thunderbolt Bus 2):​

1,3​

1,4​

2,3​

2,4​

Did you try using the Top Thunderbolt Bus and/or I/O Card Thunderbolt Bus (one per bus)?

You can try mapping Thunderbolt ports to AMD GPU ports using the AGDCDiagnose command (use a 4K display since XDR uses two ports at a time). You have 12 AMD ports, 6 per GPU, 5 used by the GPU connected to the HDMI port, 4 used by the other GPU. See how the behavior changes when the other GPU is removed.

https://www.apple.com/mac-pro/pdf/Mac_Pro_White_Paper_Feb_2020.pdf

For a working 6K signal, you should see dual HBR3 connections from a single AMD GPU to an XDR display in the AGDCDiagnose output.