MP All Models Mac Pro 2019 - PCIe Expansion Opportunities?

[alanmadzar]

Cross sync isn't going to work if on two different PCI-e bus bundle pairs. The two slots will be at two different addresses. There may be some Rube Goldberg gyrations someone could go through, but it won't buy anything significant to merit a market for cards.

1. As pointed out in post #16 above the Apple configurator suggests that the last are not all on the same switch as slot 5. For the ones on the same switch as slot 5 there is no "cross sync" coming since pulling on the same source. For the one apparently on a different switch ( slot #7 )

2. if wanted to take two x8 storage controllers and make it look like an approximately x16 worth of bandwidth then something like SoftRaid ( or some other software RAID) would work far more easier and be far more flexible with whatever other chokepoints the two MPX modules put on the system overall bandwidth.



Metal is more a library ( a set of calls that programs use ) than program itself. Technically it isn't really Metal that is doing the Afterburner work. It is another library further up the stack. That library may make some use of Metal calls to transfer data if there is some direct memory access transfers of data between cards.

Metal also isn't completely CPU usage free either.

.


With Afterburner and the GPGPU picking up lots of the computational grunt work then yes there should be enough CPU cycles around in most workloads for a Software RAID not to pose that much of an impact.

Probably is a normal context for Afterburner as it would be tough to keep that card fed with 3x 8K RAW data streams from the x4 PCI-e v3 bandwidth T2 internal drive. Or the two SATA connectors. Need something incrementally better than x8 PCI-e v3 to get that much RAW data streaming in parallel ( 8K 10-bit color , HDR , 24 fps ) .




just need the aggregate bandwidth. Not everything has to come from the same physical drive.

So... will a fully configured Mac Pro 2019 (2 MPX modules + Afterburner) take up all 64 PCIe lanes provided from the CPU? It looks like that second port that connects to the left of the PCIe slots is going to take a bandwidth of x8. So 2 x (16 + 8) = 48. Plus Afterburner - 48 + 16 = 64?

 

[deconstruct60]

So... will a fully configured Mac Pro 2019 (2 MPX modules + Afterburner) take up all 64 PCIe lanes provided from the CPU? It looks like that second port that connects to the left of the PCIe slots is going to take a bandwidth of x8. So 2 x (16 + 8) = 48. Plus Afterburner - 48 + 16 = 64?

Two 580X MPX modules won't. ( those are half width MPX modules and the two x8 slots are free and could be used for a storage connection card(s) )

Even two Vega II modules doesn't necessarily. ( e.g, if only hook up 2-3 displays to the TB ports then not really using the PCI-e bandwidth downstream from the switches that are also feeding some of the higher number slots. ).

But yes if you hooked up 8 NVMe SSDs in external enclosures to the two Vega II cards and pulled data for something else while also trying to feed the Afterburner card with three 8K RAW streams from 6-7 slots ... you'd run out of bandwidth. However, if you used those 8 SSDs to feed the Afterburner you probably won't.

If you use the MPX bandwidth to fight against the Aferburner cards bandwidth needs then will have a problem. However, that card probably costs a very substantial amount of money. If bought one then probably going to orient the primary data stream flow to keep it 'well fed' as that way earn the money back for paying for it.

Also remember that the PCI-e bandwidth goes in two directions. The primarily purpose of sending very bulky 'RAW' data to the Afterburner card is to make it relatively smaller and far easier to handle. Afterburner only gets a bit limited when doing maximum concurrent compression. Two or one stream then you don't have a much of a bottleneck. (may not need two full size Vega II MPX modules either. )

 

[alanmadzar]

Two 580X MPX modules won't. ( those are half width MPX modules and the two x8 slots are free and could be used for a storage connection card(s) )

Even two Vega II modules doesn't necessarily. ( e.g, if only hook up 2-3 displays to the TB ports then not really using the PCI-e bandwidth downstream from the switches that are also feeding some of the higher number slots. ).

But yes if you hooked up 8 NVMe SSDs in external enclosures to the two Vega II cards and pulled data for something else while also trying to feed the Afterburner card with three 8K RAW streams from 6-7 slots ... you'd run out of bandwidth. However, if you used those 8 SSDs to feed the Afterburner you probably won't.

If you use the MPX bandwidth to fight against the Aferburner cards bandwidth needs then will have a problem. However, that card probably costs a very substantial amount of money. If bought one then probably going to orient the primary data stream flow to keep it 'well fed' as that way earn the money back for paying for it.

Also remember that the PCI-e bandwidth goes in two directions. The primarily purpose of sending very bulky 'RAW' data to the Afterburner card is to make it relatively smaller and far easier to handle. Afterburner only gets a bit limited when doing maximum concurrent compression. Two or one stream then you don't have a much of a bottleneck. (may not need two full size Vega II MPX modules either. )

Thank you for your response. I also noticed that the MPX modules have thunderbolt ports in them. Is this integrated so that developers can directly integrate their work through the MPX modules?

I am trying to understand the work flow of the (Mac Pro - MPX Modules - Afterburner - Metal - Pro App Developers).

 

[deconstruct60]

Thank you for your response. I also noticed that the MPX modules have thunderbolt ports in them. Is this integrated so that developers can directly integrate their work through the MPX modules?

I don’t think so. Two things. First, The full size MPX modules take up 4 standard slots widths . So two are 8 slots wide . These modules are soaking up tons of space . There are zero native bays for HDDs . Apple didn’t 100% let go of the notion in the other recent Macs that external disks are part of the solution matrix for many . It isn’t the only path ,but it is an option . For example slots 5,6, and or 7 have SSDs installed and the Thunderbolt ports are path out to direct attached storage as the secondary storage tier .


Second, longer term going to have Thunderbolt like ports on graphics cards anyway . DisplayPort v2 will use a variant of TB v3 that is unidirectional outbound . ( it won’t necessarily need the PCIe bidirectional links for DP v2 mode but Alternate mode to that may ) .


If your are trying to infer some workload benefit because the GPU is physically close to the TB controllers , there is nothing particularly special there . Routing on DPv1.4 out the GPU to TB controller is incrementally easier , but that isn’t a big workload enabler .


There is also ability to use these as USB ports too. Not necessarily extremely high bandwidth .
Older Mac Pro had 4 FireWire , 7 USB 2 ( if count keyboard ) . Folks with a bucket of stuff to connect have options if don’t use slots 5-8 for this . In short it gives the new system a port count war checkbox item .

I am trying to understand the work flow of the (Mac Pro - MPX Modules - Afterburner - Metal - Pro App Developers).

I think they are aiming at the ‘Kitchen sink’ of workflows ( who happen to have buckets of money lying around or ‘bill it and spend other people’s money ‘) as much as anything specific with the general features .


Afterburner is a bit different. It turns RAW video into ProRes/ProResRAW. For workflows that don’t consume those it isn’t useful much at all. It would be a wasted slot and expense in that context .

 

[alanmadzar]

I don’t think so. Two things. First, The full size MPX modules take up 4 standard slots widths . So two are 8 slots wide . These modules are soaking up tons of space . There are zero native bays for HDDs . Apple didn’t 100% let go of the notion in the other recent Macs that external disks are part of the solution matrix for many . It isn’t the only path ,but it is an option . For example slots 5,6, and or 7 have SSDs installed and the Thunderbolt ports are path out to direct attached storage as the secondary storage tier .


Second, longer term going to have Thunderbolt like ports on graphics cards anyway . DisplayPort v2 will use a variant of TB v3 that is unidirectional outbound . ( it won’t necessarily need the PCIe bidirectional links for DP v2 mode but Alternate mode to that may ) .


If your are trying to infer some workload benefit because the GPU is physically close to the TB controllers , there is nothing particularly special there . Routing on DPv1.4 out the GPU to TB controller is incrementally easier , but that isn’t a big workload enabler .


There is also ability to use these as USB ports too. Not necessarily extremely high bandwidth .
Older Mac Pro had 4 FireWire , 7 USB 2 ( if count keyboard ) . Folks with a bucket of stuff to connect have options if don’t use slots 5-8 for this . In short it gives the new system a port count war checkbox item .





I think they are aiming at the ‘Kitchen sink’ of workflows ( who happen to have buckets of money lying around or ‘bill it and spend other people’s money ‘) as much as anything specific with the general features .


Afterburner is a bit different. It turns RAW video into ProRes/ProResRAW. For workflows that don’t consume those it isn’t useful much at all. It would be a wasted slot and expense in that context .

Right, but in terms of how the system works collectively, could you provide an example of how a video process flow would integrate through this system?

Like... Raw Video > Mac Pro 2019 I/O > CPU > PCIe > ( Metal (MPX Module GPU + Afterburner) ) > Output/Display

Is my logic/thinking right here?

 

[s66]

x

 

[deconstruct60]

....
The images suggest it might be M.2 modules - will be interesting to see 3rd party options for these slots.

It is not suggestive at all if have looked at the internals of an iMac Pro.

These ( Step 8 of IFixit teardown )

are not M.2 modules.

There are subcomponents of the SSD; not SSDs themselves. The T2 contains the SSD controller. It is highly unlikely that 3rd parties will sell solutions for "internals" of an Apple SSD ( just like for the intenals of other folks' SSDs. ) . The Mac Pro gets a 'black' color scheme, but probably same electronics design.

The Mac Pro is probably based on the same baseline tech as iMac Pro. For that system the 'logical' , effective SSD has 2-3 physical parts (usually 3 . Mac Pro has a skimpy on NAND card option). The Flash NAND chips are actually held on a separate daughtercards (along with what is probably a very simple communications buffer chip since NAND chips are farther away from controller than usual. ) . This likely makes failed NAND chip replacements easier. It also lets folks who have mandated physical destroy directives for retired systems to just shred these daughtercards ( instead of the whole motherboard). What it doesn't do is create a viable 3rd party replacement market.

 

[ActionableMango]

How many trash cans do you think were bought for non-technical pointy-haired managers who wanted something 'sexy'?

Hahaha, yes. The only person I know to get a trash can is a director who didn't know what it was. He only knew that it was the most expensive Mac that his employer authorized for director-level employees.

I admit the 6,1 looks great.

 

[OkiRun]

It is not suggestive at all if have looked at the internals of an iMac Pro.

These ( Step 8 of IFixit teardown )

are not M.2 modules.

There are subcomponents of the SSD; not SSDs themselves. The T2 contains the SSD controller. It is highly unlikely that 3rd parties will sell solutions for "internals" of an Apple SSD ( just like for the intenals of other folks' SSDs. ) . The Mac Pro gets a 'black' color scheme, but probably same electronics design.

The Mac Pro is probably based on the same baseline tech as iMac Pro. For that system the 'logical' , effective SSD has 2-3 physical parts (usually 3 . Mac Pro has a skimpy on NAND card option). The Flash NAND chips are actually held on a separate daughtercards (along with what is probably a very simple communications buffer chip since NAND chips are farther away from controller than usual. ) . This likely makes failed NAND chip replacements easier. It also lets folks who have mandated physical destroy directives for retired systems to just shred these daughtercards ( instead of the whole motherboard). What it doesn't do is create a viable 3rd party replacement market.

If it's true that once the decision is made on the Mac Pro 7.1 SSD configuration and there is no going back - but only PCI expansion... it really puts pressure to max out the initial SSD configuration of 8Tb from a fiscal point of view. In fact, I wouldn't mind an even higher configuration size if I'm going to be married to it for the life of the computer.

 

[s66]

x

 

[deconstruct60]

If it's true that once the decision is made on the Mac Pro 7.1 SSD configuration and there is no going back - but only PCI expansion... it really puts pressure to max out the initial SSD configuration of 8Tb from a fiscal point of view.

Not really. First, there are SATA options. There is a trade off if have a CPU that bleeding gobs of heat at the designated cheap SATA storage area, but it is an option. Second, the "pressure" thing arises more so from folks who have a deep need to pile all of their data into one single, quite large drive. There is little real "need" to do that, so it isn't really deep pressure.

Nobody is really being herded into 8TB Apple SSD drives. ( or 4TB either).

In fact, I wouldn't mind an even higher configuration size if I'm going to be married to it for the life of the computer.

How valuable is your data? Data that is stilling on a RAID 1, 4 , 5, or 6 mode set up is immune to a single drive failure. Apple's default boot drive isn't trying to solve that problem. They aren't trying super hard to do large bulk storage out of the box either. ( highly likely no BTO configuration from Apple with a HDD shipped inside the box. )

There are multiple ways to add or provision bulk storage capacity to the new Mac Pro. Apple is more so letting folks just do what they want. The notion that users are 'trapped' forever by whatever they select on the BTO page before they hit the buy button is loopy. They just have to make up their own mind. Apple isn't going to do it for you. They'll gladly take a large wad of your money if you just grow the SSD capacity toe "maximum"... practically nobody "has to" do that .

If there is a drive failure you aren't married to the initial set of NAND chips. Whether Apple offers upgrade later or not as a service time will tell. There are going to be enough alternative paths to higher capacity that they probably won't. ( probably heavily depends upon how many of these they sell and what service options Apple is looking at long term. )
[automerge]1574316310[/automerge]

The iMac (pro) can't be (easily) upgraded, but that's the glued shut machine, not a machine designed to be modular. Once Apple has a configurator for the machines online and/or more detailed specs of the components and/or the spare modules to fit in existing 2019 mac pros, it'll all become more clear (and also become clear just how expensive it's all going to be)

It is more complicated than that because it isn't just about "parts". A replaced NAND daughter card probably needs to be revalidated/authenticated to the T2 chip. The SSD is basically going to have to be 'reset' back to a scratch initial state. Removing a card isn't removing a whole SSD. It just pragmatically removes a sub component ( fraction) of a SSD. It is going to have to be reinitialized. And that highly likely isn't just a "part" since an intricate part of the baseline system integrity.

Also not going to be able to "mix and match" NAND blades. The wear usage is metadata probably gets screwed up if simply just remove one. Even if it wasn't don't want two substantively different wear patterns on the NAND chips collections.

Whether the case is sealed closed or open the T2 still has the same security requirements and role to fill. The Mac Pro case coming off probably gets no huge exception from that role or requirements. There is no point of putting in there if just going to subvert it.

 

[thisisnotmyname]

How valuable is your data? Data that is stilling on a RAID 1, 4 , 5, or 6 mode set up is immune to a single drive failure. Apple's default boot drive isn't trying to solve that problem. They aren't trying super hard to do large bulk storage out of the box either. ( highly likely no BTO configuration from Apple with a HDD shipped inside the box. )

RAID 4? Who's running that today?

Just a friendly reminder to anyone reading this that despite the bit about mirror and parity RAID levels being immune to a single drive failure, RAID is not backup. Use a true backup strategy, preferably geographically diverse as well, in addition to RAID. RAID does not protect you from fire or theft. RAID does not protect you from user error. RAID does not protect you if you have more disks fail than you have mirror or parity.

Also keep in mind that when you lose one drive there can be a statistically significant chance you'll lose a second while rebuilding the array and the whole array will die. RAID 5 on large arrays isn't a good idea because of this fact.