New Mac Pro - two internal flash drives

[magicMac]

Hi,

I noticed the new mac pro with two GPUs has a single internal PCIe flash drive over one of the GPU's and a blank space over the other one.

Just out of curiosity, does anyone know if it's possible to take this GPU/storage combo board of one mac pro and install it as the secondary in another mac pro in order to double it's internal max storage to 2TB of PCIe flash?

Jonathan

 

[AidenShaw]

Hi,

I noticed the new mac pro with two GPUs has a single internal PCIe flash drive over one of the GPU's and a blank space over the other one.

Just out of curiosity, does anyone know if it's possible to take this GPU/storage combo board of one mac pro and install it as the secondary in another mac pro in order to double it's internal max storage to 2TB of PCIe flash?

Jonathan

No, it's not possible. Look at the power connectors to the GPU daughtercards (the circles at top left and top right) - they're mirror images. You can't put two right GPUs into the tube - you need a left and a right.

 

[flowrider]

^^^^Never thought of this before, but your answer begs the question - What would happen if you took two nMPs one with say two D500s and one with two D300s, and switched one of the two GPUs from one machine to the other

Lou

 

[artherd]

Actually those PSU connector bus-bars look somewhat easy to re-route with enough work (maybe replacing them entirely with heavy gauge wire). Unclear though: are the PCIe connectors the same? I doubt it, as I suspect the 2nd GPU w SSD actually contains 2 PCIe "connectors" (i.e. lanes) either that or it uses a switch or some other hack (maybe 8 lanes to GPU and 4 to PCIe flash instead of 16 to GPU) to split things out.

 

[Dreadnought]

^^^^Never thought of this before, but your answer begs the question - What would happen if you took two nMPs one with say two D500s and one with two D300s, and switched one of the two GPUs from one machine to the other

Lou

I got burned 6 months ago asking a similar question; you can't install two gpu's with both a ssd or both the none ssd. As said above you'll need a left and right as the cards are mirrored, probably also the connectors. I don't know what will happen when you use two different cards in there, so swapping the left pr the right. Probably it will work, maybe in the software there is some code that prevents it.

 

[deconstruct60]

....
Unclear though: are the PCIe connectors the same? I doubt it, as I suspect the 2nd GPU w SSD actually contains 2 PCIe "connectors" (i.e. lanes) either that or it uses a switch or some other hack (maybe 8 lanes to GPU and 4 to PCIe flash instead of 16 to GPU) to split things out.

This image is suspected to come from a service manual.

http://i.imgur.com/ItIqxDY.png

Anandtech's probing produced a similar mapping

http://www.anandtech.com/show/7603/mac-pro-review-late-2013/8

[ The difference is largely the placement of the discrete USB 3.0 controller. And Aantech's math on the switch is correct. Not sure what bandwidth a USB 3.0 controller is getting if fully load all three TB controllers. connecting to PCH avoids that problem. ]


The problem is there are no substantive PCIe lanes (and bandwidth) for a second SSD to connect to. Apple loads all of the display load onto one GPU (so harder to "short change" it on bandwidth) and trimming bandwidth off of a compute GPGPU generally makes even less sense if looking to provide top performance.

The PCH/chipset is max out. Even if they could choke off and de-duplicate Ethernet and squeeze in a x2 PCIe SSD (from MBPs) but it would lower bandwidth than the other one. ( PCH typically goes 4 x1 + 1 x4 or 8 x1 ).


What Apple needs is either a PCH/chipset that has more PCIe v2 lanes (4 more ) or has eight v3 lanes In the first case, for the Mac Pro the SATA lanes the PCH provides are useless and the USB lanes are largely same way. Apple (and others ) may convince Intel they need another variant in their chipset line-up. In the second case, some of those eight lanes can be split ( x4 PCIe v3 switched into two x4 PCIe v2 lanes bundles ) and/or the SSDs upgraded to PCIe v3 so demand less lanes ( just x2 SSDs ) .

The next generation PCH chipset ( 610 series ) has USB 3.0. If believe the diagram then could propose to tap the "extra" x4 on the I/O board switch when the discrete USB 3.0 controller gets absorbed. Looks like a couple of potential problems there both short and long term.

A. hugely unequal trace lengths. The TB controllers are co-located with switch while the SSD line would have to bounce down to backplane and up to card.

B. Apple may punt for a long time on USB 3.1 but that's another multi lane consumer coming down the pike.

C. Thunderbolt is potentially going to blow past the total PCIe v3 budget also if it switches to the newer bandwidth.

 

[magicMac]

No, it's not possible. Look at the power connectors to the GPU daughtercards (the circles at top left and top right) - they're mirror images. You can't put two right GPUs into the tube - you need a left and a right.

i hadn't noticed that! hopefully apple/intel come to their senses and create more PCIe lanes to allow a 2nd internal flash drive in addition to the two GPUs. It looks like the chassis and layout planned for it but perhaps the chipset doesn't allow this amount of PCIe lanes at the moment.

The other option might be a bigger flash storage board, at the moment it's only a little wider than the connector but quite tall. Perhaps if they made it much wider, it could fit 2,3 or 4 times as many NAND chips on it to increase storage.

 

[deconstruct60]

i hadn't noticed that! hopefully apple/intel come to their senses and create more PCIe lanes to allow a 2nd internal flash drive in addition to the two GPUs.

It would be nice to have 10GbE , USB 3.1 , and PCIe v3 Thunderbolt too. It isn't just the SSD with upward bandwidth pressure. Apple appears disconnected from Intel in rapidly dropping SATA. If Apple had left room for at least one SATA device this capacity shortage (need two drives) wouldn't be as pronounced a problem.

A very simple solution would be just a SATA based SSD on the second GPU card and route that to the vastly under utilized SATA subsystem on the PCH. It isn't a symmetric solution so I doubt Apple's OCD design team could deal with it but it would work.

It looks like the chassis and layout planned for it but perhaps the chipset doesn't allow this amount of PCIe lanes at the moment.

More so it looks like Apple took the core basic design for one of the boards and just tweaked it slightly. Move some power handling subsystem and just "delete" the SSD infrastructure. Cut/Paste/Rotate on the design. It is a cheaper path to second card.

The other option might be a bigger flash storage board, at the moment it's only a little wider than the connector but quite tall. Perhaps if they made it much wider, it could fit 2,3 or 4 times as many NAND chips on it to increase storage.

Density of NAND chips goes up over time. Either small process or multiple bits per cell. Also can stack layers into slightly thicker packages. More likely they'll just wait.

 

[AidenShaw]

i hadn't noticed that! hopefully apple/intel come to their senses and create more PCIe lanes to allow a 2nd internal flash drive in addition to the two GPUs.

You don't need more PCIe lanes - the MP6,1 has 4 unused PCIe 2.0 lanes.

And - you don't need to have every PCIe lane on a full bandwidth pipe to the CPU.

It's entirely reasonable to use PCIe switches to over-commit PCIe lanes. Many people would gladly take a second 1 TB internal drive - even knowing that it shares the bandwidth with the first drive.

 

[deconstruct60]

You don't need more PCIe lanes - the MP6,1 has 4 unused PCIe 2.0 lanes.

Two boards away from the GPU board. And no bandwidth if load up the TB controllers.

And - you don't need to have every PCIe lane on a full bandwidth pipe to the CPU.

The SSD is hooked to the PCH already.

It's entirely reasonable to use PCIe switches to over-commit PCIe lanes.

It is reasonble to overcommit with low utilization devices. Bluetooth share a link with some low bandwidth audio chip. Sure.

Reasonable to overcommit with devices that attempt to completely saturate their physical link allocation (e.g., 1200MB/s SSD )? Not really. Almost every PC design that overcommits largely either does so on secondary, low bandwidth deivces ( a legacy PCI slot , low bandwith like Bluetooth 3.0 or lower) or chops down the bandwidth to the device ( push GPU card into x8 ) and partially makes up chopping down bandwidth by splitting the load to multiple devices (along with in most cases adding another backhaul network ; card-to-card Crossfire/SLI ).

There generally are no overcommitted GPGPU HPC cards. Nor are there overcommitted low latency I/O cards.

Many people would gladly take a second 1 TB internal drive - even knowing that it shares the bandwidth with the first drive.

So if put the two drives in a RAID 0 configuration the new set up is just as fast as a single drive. I think that is going to draw as many folks who think that is bogus as think it is great systems design.

Anymore than will find alot of fans of multiple port SATA controllers than only offer one drive's worth of combined bandwidth. Hook a 6Gb/s drive to the controller get 6Gb/s. Hook a second drive and get 3Gb/s .... don't see many controllers offered that do that. On the 4th, 5th , 6th drive perhaps put on the just one more on a multiple thousand dollar system Most folks want over 1 drives worth of backhaul out of the SATA controller. It isn't going to be any different in PCI-e land.

 

[AidenShaw]

Two boards away from the GPU board. And no bandwidth if load up the TB controllers.

Then put the 2nd SSD two boards away.

Since the dawn of time computer systems have been oversubscribing bus bandwidth. It's a very good thing - you have connectivity for storage or network or other devices - with the knowledge that you can't run every single device at full bandwidth simultaneously. If you need max bandwidth, then you don't connect everything. If you need double the storage, the you add the second PCIe blade and accept that it doesn't have the raw bandwidth of the first one.

The Apple laptops and all-in-ones deal with this, and the earlier Mac Pros.

But for some reason, some of the Apple fans think that every device on the MP6,1 has to have a full bandwidth reserved pipeline to the CPU plex.

 

[alphaod]

Well hopefully Apple will come to their senses and introduce two SSD connectors in their next Mac Pro.

 

[magicMac]

the move from SATA to PCIe makes perfect sense, what with no SATA bottleneck and no differentiation between enterprise and consumer any more but the fact Apple haven't used the M.2 NGFF standard for the socket and board size makes the whole thing a bit off putting personally and makes SATA still seem like a more attractive option due to upgradability despite the lower performance.

If you got a 1TB MacPro or MacBook Pro today in the knowledge than in 2-3 years you could take that flash drive out and put a 2,3 or 4TB flash drive in it's place, then it's going to be a much better investment.

Just like CPUs, I'm sure NAND will indeed get smaller in chip size and larger in capacity for the same chip size, but who out there is selling flash drives with Apple's PCIe storage connector? Even apple won't sell you the part to upgrade the storage after purchase. Although there are some of the 1TB ones on eBay (presumably spares from faulty machines).

Having two PCIe storage slots would certainly make the investment seem more worth in terms of eventually moving away from external mechanical drives - quite important for laptops.

 

[artherd]

2 pcie SSDs from Apple just isn't going to happen. Ever. Their mantra is "If you need more than 1TB of SSD, go thunderbolt".

 

[mcnallym]

If you look at the Apple range then the whole range now with the (possible ) exception of the mac mini server is standard on one storage volume internally.

iMac = 1 HDD, 1SSD or 1 Fusion Drive ( 1 SSD and 1 HDD presented as one volume)
MBA = 1 SSD
MBPro = 1 SSD
nMP = 1 SSD
mini - 1 HDD, 1 SSD or 1 Fusion Drive
mini Server = I believe that comes with the two drives in a RAID so one volume presented internally

Yes I know that can buy adaptors to allow fitting two drives in the mini, and if buy a Fusion iMac then can rebuild or even strip out the HDD and replace with an SSD, but am on about "as taken out of the box"