Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
MP 7,1 Allocating PCI Pools?
- Thread starter choreo
- Start date
- Sort by reaction score
If that's the result of automatic bandwidth configuration, then it seems a little strange since there's room on Pool B to move the FireWire card.
It doesn't really matter since you usually won't be using all devices in Pool A at once (four NVMe devices on the Sonnet 4x4 and the FireWire card). Are the NVMe devices in a RAID 0? If not, then they are definitely not going to be used at the same time.
Is the Apple I/O card able to use Pool A? If so, then that would contradict Apple's Mac Pro white paper that says "all Thunderbolt ports in the systems are fixed to use root port B". Each Thunderbolt controller is PCIe 3.0 x4 (but can't use much more than 23 Gbps of bandwidth). The W5700X card has two Thunderbolt controllers.
The diagram doesn't show how many lanes the cards are using or their link rate (it only shows the max number of lanes).
Post results from
The diagram doesn't show anything connected to slot 6 even though you say there's a Sonnet Tempo there.
I see that you meant the Sonnet Tempo is in slot 5 with a 2.5" SSD (one or two?)
It doesn't really matter since you usually won't be using all devices in Pool A at once (four NVMe devices on the Sonnet 4x4 and the FireWire card). Are the NVMe devices in a RAID 0? If not, then they are definitely not going to be used at the same time.
Is the Apple I/O card able to use Pool A? If so, then that would contradict Apple's Mac Pro white paper that says "all Thunderbolt ports in the systems are fixed to use root port B". Each Thunderbolt controller is PCIe 3.0 x4 (but can't use much more than 23 Gbps of bandwidth). The W5700X card has two Thunderbolt controllers.
The diagram doesn't show how many lanes the cards are using or their link rate (it only shows the max number of lanes).
Post results from
system_profiler SPPCIDataTypeI see that you meant the Sonnet Tempo is in slot 5 with a 2.5" SSD (one or two?)
Yes, I was dumb enough to buy in to the last Apple connectivity standard that was going to be the end-all! Have about 25 external Firewire 800 drives with various clients files from the past 30 years that still work perfectly fine. My 2012 MacPro that I have been using until now (since that was the last modular Mac Apple offered) had built-in Firewire 800 ports AND there was no way to connect the latest Thunderbolt 3 (this week's standard) on that machine anyway. Even if I bought a new external TB3 storage device now, it would not gain me much with those legacy files which are mostly archived at this point.
[automerge]1595121408[/automerge]
Yes the Sonnet card in slot 4 has (4) 1TB Samsung 970 EVO Plus NVMe modules in RAID O.
Yes the Sonnet card in slot 5 has a single 1TB Samsung SSD for a Boot Drive backup which CCC clones to once a day.
I guess the easiest thing to do would be just switch the Firewire card in slot 7 to Pool B?
Last edited:
It won't change anything noticeably (FireWire is only 800 Mb/s which is like one tenth of a single PCIe lane) but does satisfy my OCD.
Maybe there's other considerations that we're not seeing (interrupts? device count? ...)
Send Apple feedback for this situation.
The Tempo is not an x16 card. It should go in slot 6. Then save slot 5 for a future x16 card.
Move the M2 4x4 to slot 3 if you're not going to add a second GPU. It will improve performance very slightly (removes PCIe switch between CPU and M2 4x4). It will also completely free up Pool A. Then you can move the FireWire card to Pool A to even out the devices.
I will do as you suggest. The only reason I had the M2 4x4 in Slot 4 was to keep it from laying right on top of the video card for air circulation - looks pretty tight?
I wouldn't worry about it - there's enough cooling on the business side of the M2 4x4 (component side which is the side containing the fan). And in fact you'll be giving more room for the M2 4x4 fan. The card is full length which means it's supported at both ends so it won't collide with the MPX module. The MPX module has air movement internally for cooling - all the heat generating components are on the slot 1 side instead of the slot 2/3 side.
I moved the Tempo Card to Slot 6 and the M2 4x4 to Slot 3 (as you suggested). Here is the new "AUTOMATIC" configuration. Not sure why such a big difference, but looks good on paper? Where did the extra 100% that the M2 card had when in slot 4 vanish to?
Last edited:
Looks fine.
The difference between now and before for PoolA is that you moved the x16 card out of Pool A to Slot 3. Slot 1 and Slot 3 are like their own separate pools. There are four x16 connections to the CPU:
Slot 1
Slot 3
Pool A
Pool B
Slot 1 and Slot 3 have 16 fixed lanes each and are directly connected to the CPU. There is no sharing for these slots.
There are 64 PCIe lanes to divide between Pool A and Pool B. All the devices in Pool A share an x16 link to the CPU. All the devices in Pool B share a different x16 link to the CPU.
The difference between Pool A and Pool B in your current setup is that Pool B has higher bandwidth devices connected (The I/O card is PCIe 3.0 x4, the Thunderbolt controllers of the W5700X are also PCIe 3.0 x4 so maybe it should say Pool B is higher than 50%?). The FireWire device is something like PCIe 1.0 x1 and the Tempo is like PCIe 3.0 x1.
Thanks for the detailed and clear explanation! I obviously have been out of the technical side too long and have not educated myself enough on this new architecture. Learning!
After some thinking : you have four Thunderbolt 3 controllers with no Thunderbolt devices connected.
Each Thunderbolt controller has a Thunderbolt NHI device (for Thunderbolt networking, etc) and a USB 3.1 gen 2 controller for USB devices that you might connect to the Thunderbolt ports. These devices report themselves as PCIe 1.0 x4 (even though they have higher bandwidth than that).
So PCIe 1.0 x4 per device times two devices per controller, times four controllers = PCIe 1.0 x32 = PCIe 2.0 x16 = PCIe 3.0 x8 = 50% of Pool B. This can be tested wth adding/removing GPUs and the I/O card and Thunderbolt devices.
The PCIe 3.0 x4 connection that a Thunderbolt controller has (PCIe upstream bridge) is not used in the pool calculation. In general, PCIe bridges are not included in the pool calculation - only PCIe devices are included (PCIe bridges are the pipes while PCIe devices provide the water in the pipes).
That's my theory anyway.
Each Thunderbolt controller has a Thunderbolt NHI device (for Thunderbolt networking, etc) and a USB 3.1 gen 2 controller for USB devices that you might connect to the Thunderbolt ports. These devices report themselves as PCIe 1.0 x4 (even though they have higher bandwidth than that).
So PCIe 1.0 x4 per device times two devices per controller, times four controllers = PCIe 1.0 x32 = PCIe 2.0 x16 = PCIe 3.0 x8 = 50% of Pool B. This can be tested wth adding/removing GPUs and the I/O card and Thunderbolt devices.
The PCIe 3.0 x4 connection that a Thunderbolt controller has (PCIe upstream bridge) is not used in the pool calculation. In general, PCIe bridges are not included in the pool calculation - only PCIe devices are included (PCIe bridges are the pipes while PCIe devices provide the water in the pipes).
That's my theory anyway.
I do have devices connected to the 4 Thunderbolt3 connections on the W5700X card. (3) 27" QHD Monitors and a Caldigit Hub on the 4th port (mainly to give me Digital Audio out for speaker system). I am not currently using the two TB3 connectors on the I/O card or the top case. Nor am I using the TB3 connector out on the Hub.
Are the devices USB or Thunderbolt? Do any PCI devices exist in the PCI section of System Information.app in Thunderbolt slots?
On my setup I'm getting 125% in Pool B.
From reading the forum I'm not even sure if this is a "bad" thing or just an artifact of how the things are configured internally by the 7,1
Should Itry and cognate this?
I have a OWC 4M2 in Slot 7 and a Sonnet M2 (fanless version) in Slot 5.
From reading the forum I'm not even sure if this is a "bad" thing or just an artifact of how the things are configured internally by the 7,1
Should Itry and cognate this?
I have a OWC 4M2 in Slot 7 and a Sonnet M2 (fanless version) in Slot 5.
Sonnet M2 is x16, move it to slot 3 into it's own pool (not A or B)
OWC Accelsior 4M2 is an x8 card so keep it in slot 7 but move it to pool A.
What's the GPU in slot 1? A GPU can't have more than two Thunderbolt controllers so I'm wondering where the extra 25% is coming from.
What do you have connected to Thunderbolt/PCI?
system_profiler SPPCIDataType SPThunderboltDataTypeI only see three Thunderbolt controllers (201,202; 30,31; 87,88) - maybe one of them is powered down because it has nothing connected? (Thunderbolt controllers are: two on the W5700X, one on the top of the Mac Pro, one on the I/O card).
In that case, that would be PCIe 1.0 x4 * 6 = PCIe 1.0 * 24 = PCIe 1.0 * 12 = PCIe 3.0 * 6; 6/16 = 37.5%
The CalDigit TS3 Plus has 4 PCIe devices (58,59,60,61) which are PCIe 1.0 x1 (guessing for this one), PCIe 3.0 x1, PCIe 2.0 x1, PCIe 2.0 x1 = PCIe 3.0 * 2.25 = 14%
37.5 + 14 = 51.56%
The Caldigit's connection to the Thunderbolt controller (probably the second port of the controller at 30,31) is identified as PCIe 1.0 x4 which would limit the bandwidth to approximately PCIe 3.0 x1 = 6.25% (remember that all downstream devices of a Thunderbolt controller - XHCI, NHI and the two ports) have their link rate identified as PCIe 1.0 x4 even though they have higher bandwidth than that).
37.5 + 6.25 = 43.75%
The math is missing something... Maybe the pool calculation doesn't count Thunderbolt devices? Maybe there's some other device counted with Pool B?
Thanks for that, I'll move them around - in the middle of a project right now so it'll have ton wait until that's done
1 is A Radeon W5700X, other things connected via Thunderbolt are two Dell 4k/27" Display's, an older Promise Pegasus R4 (Thunderbolt 1) RAID setup and connected to that an older LaCie RAID sew tip for TimeMachine, which is also Thunderbolt 1. And lastly, a OWC dock for various USB connections (flooring drives, charging iPhone, iPad etc).
Followed your suggestions and now have proper allocation. See attached.
Thanks for that, I appreciate your time and input.
Btw, exactly what does 125% Allocation actually mean as far as performance goes?
100% is PCIe 3.0 x16.
125% means you have devices that add up to more than PCIe 3.0 x16 - something like PCIe 3.0 x20 - if they were all used at once (which rarely happens) then performance for some of them would drop.
Remember there are 64 lanes to divide between two pools of 16 lanes.
Pool A is easy to calculate because it only consists of devices that you insert into PCIe slots.
Pool B has PCIe slots and Thunderbolt controllers.
If you do
ioreg in Terminal.app. Then you can see where the PCIe devices are.PCI0 appears to be chipset stuff which includes built-in USB, Bluetooth, SATA, serial, NVMe, T2, Audio, WiFi, Ethernet.
PC01/BR1A appears to be Slot 1.
PC02/BR2A appears to be Pool A with slots at DS##.
PC03/BR3A appears to be Slot 3.
PC04/MCP0 appears to be Pool B with slots and Thunderbolt controllers at DS##. The motherboard controllers are at DS09 (XHC4) and DS0A (XHC5). I don't know if those change if you change the pools or add a GPU that has its own Thunderbolt controllers. A Mac Pro can have 2, 4, or 6 Thunderbolt controllers.
If you do
ioreg -lw0 then you can decode IOPCIExpressLinkStatus to find their link rate and link speed.For example:
Code:
ioreg -lw0 > ioregMacPro71.txt
./ParseIORegPCILinkStatus.sh ioregMacPro71.txt > ioregparsedioregMacPro71.txtIOPCIExpressLinkStatus only contains the value at startup. The current PCIe link rate and link width changes for some devices (for example, some GPUs switch to PCIe 1.0 speed if they are not doing a lot of work). To get current link rate and link width, you need to examine the PCIe registers of each device. pcitree.sh is a script that uses pciutils to get that info.
Hello, I just bought two Sonnet M.2 4x4 SSD PCIe Card. I've placed in slot 5 the first one (It's 4x2TB Sabrent SSD) and the second one (It's 1x8TB Sabrent SSD) into slot 4 (previously I placed it into slot 6 but Expansion Slot Utility suggested to move it into slot 4). Now this is what Expansion Slot Utility. Any suggestion? I don't have any other external device connected to the mac apart the thunderbolt monitor which is connected to the AMD Radeon Pro W5700X.
Can anybody help?
Can anybody help?
