Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Status
The first post of this thread is a WikiPost and can be edited by anyone with the appropiate permissions. Your edits will be public.

tsialex

Contributor
Jun 13, 2016
13,455
13,602
Since we frequently talk about PCIe switches maybe this will be of interest to some people.

Today's iFixit teardown of the 2019 Mac Pro finally showed how MP7,1 have so many available PCIe lanes, Apple used a PLX PCIe switch PEX8798 (PLX is a Broadcom company). It's a monster of a switch.

Screen Shot 2019-12-17 at 14.20.57.png
I only found documentation for the PEX8796 version (96 lanes, 24 ports, 2 virtual switches, PCIe 3.0, 18.6W), you can download it here. Seems the PEX8798 documentation is not online yet, it's a typo by iFixit or maybe it's a part made for Apple use only. Basically, it's similar to the PEX8647 (48 lanes, 5 ports, PCIe 3.0, 8W), used on SSD7101A-1, but double the lanes/power consumption and with almost 5x more ports.
BkatMPnESpwZtDJc.huge


Btw, this eliminate any hopes of lane bifurcation partitioning support for any slots connected to the PCIe switch.
 
Last edited:

Xteec

macrumors regular
Sep 21, 2012
146
71
Australia
Since we frequently talk about PCIe switches maybe this will be of interest to some people.

Today's iFixit teardown of the 2019 Mac Pro finally showed how MP7,1 have so many available PCIe lanes, Apple used a PLX PCIe switch PEX8798 (PLX is a Broadcom company). It's a monster of a switch.

View attachment 883310
I only found documentation for the PEX8796 version (96 lanes, 24 ports, PCIe 3.0, 18.6W), you can download it here. Seems the PEX8798 documentation is not online yet or maybe it's a part made for Apple use only. Basically, it's similar to the PEX8647 (48 lanes, 5 ports, PCIe 3.0, 8W), used on SSD7101A-1, but double the lanes/power consumption and with almost 5x more ports.
BkatMPnESpwZtDJc.huge
Nice one!

I’m no expert, so can I ask does this suggest the cheaper cards like the Asus are a bit more promising because of this?
 

tsialex

Contributor
Jun 13, 2016
13,455
13,602
Nice one!

I’m no expert, so can I ask does this suggest the cheaper cards like the Asus are a bit more promising because of this?
The last glimmer of hope is if slots 1 and 3 are directly connected to the CPU and Apple enabled the lane partitioning support in the firmware. Intel developed PCI Express Lane Partitioning as a cost reduction way to eliminate a PCIe switch to share lanes, since MP7,1 already has a monster of a switch, don't make any sense for Apple enabling it.

PLX PCIe switches don’t do lane partitioning, so this eliminate adapters that need bifurcation support like the ones from the Don’t Buy list of the first post.

Microsemi has a new PCIe switch family specially designed for storage solutions that can have lanesplitting to even x2, but this family of switches is designed for high density NVMe storage servers and not real PCIe slots with pseudo-dynamic lane splitting and this is no use at all for us here, since Apple went for the real thing with the best PLX/Broadcom switches.
 
Last edited:
  • Like
Reactions: bsbeamer

joevt

macrumors 604
Jun 21, 2012
6,968
4,262
Mac Pro year models NVMe support:

Mac Pro 2006 (MP1,1): not possible, can't run Sierra/High Sierra.
Mac Pro 2007 (MP2,1): not possible, can't run Sierra/High Sierra.
Mac Pro 2008 (MP3,1): possible but risky procedure, need to inject APFS/NVMe EFI modules inside the BootROM and run 10.12/10.13*.
Mac Pro 2009 (MP4,1): use MP5,1 firmware and update to at least 140.0.0.0.0 and 10.12/10.13*.
Mac Pro 2010/2012 (MP5,1): upgrade to BootROM 140.0.0.0.0 or newer, current one is 144.0.0.0.0, and install 10.12/10.13* to have full native support.
Mac Pro 2013 (MP6,1): upgrade to BootROM MP61.0120.B00 or newer, current one is 133.0.0.0.0, and install 10.12/10.13* to have full native support

* 10.12 only supports 4Kib / sector blades while 10.13 and newer macOS releases supports both 4Kib /sector and 512 bytes / sector blades.
For the Mac Pro 2006/2007, can you explain why the NVMe solutions for 10.11 El Capitan won't work? You have to boot using a different boot loader (Piker Alpha's). I understand jimj740's NVMeGeneric.kext may have KPs as you've explained elsewhere but I think it worked ok when I built my Hackintosh back then. There's also the patches for Apple's NVMe driver (Rehabman's repository has a full set of those). If they work at all then they should be mentioned for completeness.

About MacPro3,1 and later, updating the EFI is only required for direct boot support. A third party boot loader (e.g. rEFInd) can be used to boot NVMe/APFS without modifying the ROM.


Thanks! and what controllers are cards for mac pro 1?
Socket 3 (Key M) M.2 devices are PCIe devices containing a NVMe or AHCI controller. For MacPro1,1, you would choose AHCI unless you want to go through the hoops of trying NVMe (if those hoops actually make it work). MacPro1,1 is limited to PCIe gen 1 speed ~750 MB.s. You would need a x16 card with a PCIe switch to get the full 3000 MB/s of a single PCIe M.2 drive.

Since we frequently talk about PCIe switches maybe this will be of interest to some people.

Today's iFixit teardown of the 2019 Mac Pro finally showed how MP7,1 have so many available PCIe lanes, Apple used a PLX PCIe switch PEX8798 (PLX is a Broadcom company). It's a monster of a switch.

I only found documentation for the PEX8796 version (96 lanes, 24 ports, PCIe 3.0, 18.6W), you can download it here. Seems the PEX8798 documentation is not online yet or maybe it's a part made for Apple use only. Basically, it's similar to the PEX8647 (48 lanes, 5 ports, PCIe 3.0, 8W), used on SSD7101A-1, but double the lanes/power consumption and with almost 5x more ports.

Btw, this eliminate any hopes of lane bifurcation support for any slots connected to the PCIe switch.
The picture of the PEX is unclear. The second 8 looks different than the first. It could be a 6. If it's an 8, it might mean it has more ports (but Apple doesn't need 24 ports so maybe it has less?) There is a comment on the iFixit teardown page that says the MacPro7,1 uses two x16 upstream links from the PEX, leaving 64 lanes for downstream slots. Has anyone posted an IORegistry dump? That's a topic for a different thread.
 

tsialex

Contributor
Jun 13, 2016
13,455
13,602
For the Mac Pro 2006/2007, can you explain why the NVMe solutions for 10.11 El Capitan won't work? You have to boot using a different boot loader (Piker Alpha's). I understand jimj740's NVMeGeneric.kext may have KPs as you've explained elsewhere but I think it worked ok when I built my Hackintosh back then. There's also the patches for Apple's NVMe driver (Rehabman's repository has a full set of those). If they work at all then they should be mentioned for completeness.

About MacPro3,1 and later, updating the EFI is only required for direct boot support. A third party boot loader (e.g. rEFInd) can be used to boot NVMe/APFS without modifying the ROM.



Socket 3 (Key M) M.2 devices are PCIe devices containing a NVMe or AHCI controller. For MacPro1,1, you would choose AHCI unless you want to go through the hoops of trying NVMe (if those hoops actually make it work). MacPro1,1 is limited to PCIe gen 1 speed ~750 MB.s. You would need a x16 card with a PCIe switch to get the full 3000 MB/s of a single PCIe M.2 drive.


The picture of the PEX is unclear. The second 8 looks different than the first. It could be a 6. If it's an 8, it might mean it has more ports (but Apple doesn't need 24 ports so maybe it has less?) There is a comment on the iFixit teardown page that says the MacPro7,1 uses two x16 upstream links from the PEX, leaving 64 lanes for downstream slots. Has anyone posted an IORegistry dump? That's a topic for a different thread.
Honestly, it's better to just buy an AHCI blade and forget NVMe with MP1,1 or 2,1. Even if you accept the frequent KPs, you still can't boot from the blade. It's a lot of work to get it going with unstable 3rd party kexts for just 750MB/s throughput, but feel free to add a note, or edit, that it can be make to "work".

Yes, MP3,1 can be made to work with an USB key with rEFInd and NVMe/APFS modules, someone should make an updated post about it.

I searched a lot for the PEX8798, nada. Maybe iFixit made a typo there, maybe it's a special design for Apple, this really need second source confirmation. At least we now know that M.2 adapters that require PCIe lane splitting don't work with MP7,1.
 
Last edited:

Xteec

macrumors regular
Sep 21, 2012
146
71
Australia
The last hope is if slots 1 and 3 are directly connected to the CPU and Apple enabled the lane splitting support in the firmware. Intel lane splitting was developed as a way to eliminate the cost of a PCIe switch, since MP7,1 already has a monster of a switch, don't make any sense for Apple enabling it.

PLX PCIe switches don’t do lane bifurcation splitting, so this eliminate adapters that need chipset lane splitting support like the ones from the Don’t Buy list of the first post.

Microsemi has a new PCIe switch family specially designed for storage solutions that can have lanesplitting to even x2, but this family of switches is designed for high density NVMe storage servers and not real PCIe slots with pseudo-dynamic lane splitting and this is no use at all for us here, since Apple went for the real thing with the best PLX/Broadcom switches.
Thank you!! I learned something :)
 

Parzival

macrumors regular
May 12, 2013
153
353
No support for cheap adapters that require chipset lane splitting, like the ones on the Don't Buy list of the first post. MP7,1 requires the same adapters as MP5,1.

So, please correct me if I'm wrong, but an adapter like this (which is on the list) should work with the 7,1?


And can one use multiple adapters like this in the 7,1, with blades like the Samsung 970 EVO?

Thanks!
 

tsialex

Contributor
Jun 13, 2016
13,455
13,602
So, please correct me if I'm wrong, but an adapter like this (which is on the list) should work with the 7,1?


And can one use multiple adapters like this in the 7,1, with blades like the Samsung 970 EVO?

Thanks!
Yes, it works with passive single blade adapters like kryoM.2. You can use more than one.
 
  • Like
Reactions: Parzival

tsialex

Contributor
Jun 13, 2016
13,455
13,602
I made some changes to the first post:

NVMe boot support and Mac Pro year models:

Mac Pro year model:Model Identifier:NVMe boot support:
2006 Mac Pro / Original Mac ProMP1,1not possible to inject NVMe EFI module, can't run Sierra/High Sierra. Supports natively PCIe AHCI blades.
2007 Mac Pro / Mac Pro (8-Core)MP2,1not possible to inject NVMe EFI module, can't run Sierra/High Sierra. Supports natively PCIe AHCI blades.
Early 2008 Mac ProMP3,1possible but risky procedure, need to inject APFS/NVMe EFI modules inside the BootROM and run 10.12/10.13*. Supports natively PCIe AHCI blades. Alternatively, you can also use rEFInd from another drive to bootstrap the injection of the EFI modules at boot time.
Early 2009 Mac ProMP4,1use MP5,1 firmware and update to at least 140.0.0.0.0 and 10.12/10.13*. Supports natively PCIe AHCI blades.
Mid 2010 Mac ProMP5,1upgrade to BootROM 140.0.0.0.0 or newer, current one is 144.0.0.0.0, and install 10.12/10.13* to have full native NVMe support.
Mid 2012 Mac ProMP5,1upgrade to BootROM 140.0.0.0.0 or newer, current one is 144.0.0.0.0, and install 10.12/10.13* to have full native NVMe support.
Late 2013 Mac ProMP6,1upgrade to BootROM MP61.0120.B00 or newer, current one is 133.0.0.0.0, and install 10.12/10.13* to have full native NVMe support. Needs a 12+16 adapter to use standard M.2 blades.
2019 Mac ProMP7,1native support, can boot from AHCI or NVMe blades/drives when external booting is enabled with Startup Security Utility.
* 10.12 only supports 4Kib / sector M.2 blades and U.2 drives while 10.13 and newer macOS releases supports both 4Kib /sector and 512 bytes / sector M.2 blades and U.2 drives.


Don't buy PCIe adaptors list:


Any card from ASRock/Asus/Gigabyte/MSI that don't have a PCIe 3.0 switch and requires a motherboard with PCI Express Lane Partitioning support, also know as bifurcation support, like the cards listed on the table below.

It's already confirmed that MP7,1 PCIe slots are behind a 96-lane PEX8796 PCIe switch, so no PCI Express Lane Partitioning support for 2019 Mac Pro, and the same requirements for PCIe M.2/U.2 adapters as MP5,1 are valid, the only difference is that 2019 Mac Pro slots are PCIe 3.0.

Being crystal clear, 2019 Mac Pro don't support cheap adapters that require motherboard bifurcation support.


Adapter:Motive for not working:
Aplicata Quad M.2 NVMe SSD PCIe x16 Adapterrequires motherboard PCI Express Lane Partitioning support (aka bifurcation)
ASRock Ultra Quadrequires motherboard PCI Express Lane Partitioning support (aka bifurcation)
Asus Hyper M.2 x16requires motherboard PCI Express Lane Partitioning support (aka bifurcation)
Dell Ultra-Speed Drive Quad NVMe M.2 PCIe x16 Cardrequires motherboard PCI Express Lane Partitioning support (aka bifurcation)
GIGABYTE Aorus PCIe x16 M.2requires motherboard PCI Express Lane Partitioning support (aka bifurcation)
GIGABYTE CMT2014, CMT4032 and CMT4034requires motherboard PCI Express Lane Partitioning support (aka bifurcation)
HP Z Turbo Drive Quad Prorequires motherboard PCI Express Lane Partitioning support (aka bifurcation)
MSI Xpander-Aerorequires motherboard PCI Express Lane Partitioning support (aka bifurcation)
Squid PCIe Gen 3 Carrier Board for 4 M.2 SSD modules (M-key ) (full or half-height bracket) SKU-086-B4requires motherboard PCI Express Lane Partitioning support (aka bifurcation)
SuperMicro AOC-SLG3-8E2Pfried @handheldgames Mac Pro
Synology M2D18it's a ~$200 PCIe 2.0 switch card with SATA + M2 that tops at 1500MB/s
 

chris.pielkenrood

macrumors member
Feb 24, 2018
34
3
So I finally received my Kingston A2000 NVME blade which should do speeds up to more than 2.000 mb/s.
With the KryoM.2 PCIe adapter in slot 1 I am doing "only". round 1.400 mb/s.
Is the adapter limiting the potential for the NVME Blade. I am no expert on this, so please bare with me.

Screenshot 2019-12-18 at 19.53.19.png
 

tsialex

Contributor
Jun 13, 2016
13,455
13,602
So I finally received my Kingston A2000 NVME blade which should do speeds up to more than 2.000 mb/s.
With the KryoM.2 PCIe adapter in slot 1 I am doing "only". round 1.400 mb/s.
Is the adapter limiting the potential for the NVME Blade. I am no expert on this, so please bare with me.
  1. MP5,1 is a PCIe 2.0 computer.
  2. Your NVMe blade is PCIe 3.0 x4 and will downgrade to PCIe 2.0 x4 when installed with a PCIe 2.0 computer.
  3. A x4 PCIe 2.0 slot (4 x 500MB/s) have only ~1500MB/s throughput, after all the overhead.
  4. kryoM.2 is a passive adapter and just converts the format. Only an adapter with a PCIe switch, like HighPoint SSD7101A-1 can transform a x16 PCIe 2.0 connection into a x4 PCIe 3.0.
So, your speeds are correct.
 

bsbeamer

macrumors 601
Sep 19, 2012
4,313
2,713
You are reaching the bandwidth for a single blade via single slot adapter. Speeds are entirely in line with other NVMe blades via single adapters.
 

chris.pielkenrood

macrumors member
Feb 24, 2018
34
3
Didn't knew it worked liked this. Anyway, I have a serious upgrade vs my SDD in the regular SATA drive-bay. I am happy and the KryoM.2. adapter is keeping the blade quite cool!
 

joevt

macrumors 604
Jun 21, 2012
6,968
4,262
Didn't knew it worked liked this. Anyway, I have a serious upgrade vs my SDD in the regular SATA drive-bay. I am happy and the KryoM.2. adapter is keeping the blade quite cool!
Like the internet, a PCIe bus is a series of tubes.

Some are fast and some are slow.
Some are wide and some are narrow.

An NVMe is fast (8.0 GT/s) and narrow (x4) (≈3500 MB/s).

If you connect it with a Kryo (x4) to a slow (5 GT/s) and wide (x16) classic Mac Pro slot then you get a result of slow and narrow (5.0 GT/s x4 ≈ 1500 MB/s) because there is nothing to spread out the bits over the wide x16 slot - only x4 is used.

But if you connect it with a wide card (Highpoint, Sonnet, Amfeltec, OWC, x8 or x16) that has a fast PCIe switch (8.0 GT/s), then the switch can convert the fast and narrow NVMe (8.0 GT/s x4) to your cMP slow and wide slot (5.0 GT/s x16). The switch also allows connecting multiple NVMe drives (2, 4, or 6) to the slot. Raid a few together and your slow and wide slot could get ≈ 6200 MB/s.
 

ph1l4k

Cancelled
Jul 13, 2018
4
2
As a matter of interest would anyone know if the adapter that came with my HyperX Predator M.2 PCIe SSD would work with a different SSD blade? I.e. an NVMe type from Samsung for example.
 

tsialex

Contributor
Jun 13, 2016
13,455
13,602
As a matter of interest would anyone know if the adapter that came with my HyperX Predator M.2 PCIe SSD would work with a different SSD blade? I.e. an NVMe type from Samsung for example.
Yes, no problem.
 

MacManu77

macrumors regular
Jan 31, 2018
110
4
Can I install this card in the 3rd slot x4 of MacPro 2010 right? However, I won't get the card's maximum speed.
I have the slot 1 occupied by the GPU in signature and moving it to slot 2 then I can't leave the pce card mounted with the 512GB AHCI I use now in slot 4.
Thanks.

https://www.amazon.it/CREST-si-pex4...coding=UTF8&psc=1&refRID=HTCS2D64FJ4QVQ78TD35

I update this post: I installed the link card in the slot 3 and wanted to ask if it is possible to disconnect the cooling fan and use the aluminum heat sinks for the blades or if it is better to use maybe a cable with resistance to make the fan turn a little slower.
Thanks.
 

theatwar

macrumors member
Dec 8, 2019
32
6
I just purchased two Sonnet M.2 4x4 PCIe cards for my Mac Pro 7,1 -- each card will have four 2TB Samsung 970 EVO Plus blades.

For the first card, I plan on using one of the 2TB blades as my scratch disk for Photoshop/AfterEffects and then use SoftRAID to merge the remaining three blades together.

For the second card, I plan on using one of the 2TB blades as a backup boot drive and then use SoftRAID to merge the remaining three blades together.

Quick questions:

1. Which slots should I be putting these cards in? I have the Radeon Pro Vega II in slots 1 and 2, and was planning to put the Sonnets in slots 3 and 4.

2. Which RAID option would be ideal? RAID 4 or RAID 5?

3. Do I need to 'enable trim' on these drives? Is this still done in Terminal and can someone point to the correct commands for Catalina? (Don't have any real experience in doing this, apologies if I should be asking this elsewhere).
 

thisisnotmyname

macrumors 68020
Oct 22, 2014
2,439
5,251
known but velocity indeterminate
I just purchased two Sonnet M.2 4x4 PCIe cards for my Mac Pro 7,1 -- each card will have four 2TB Samsung 970 EVO Plus blades.

For the first card, I plan on using one of the 2TB blades as my scratch disk for Photoshop/AfterEffects and then use SoftRAID to merge the remaining three blades together.

For the second card, I plan on using one of the 2TB blades as a backup boot drive and then use SoftRAID to merge the remaining three blades together.

Quick questions:

1. Which slots should I be putting these cards in? I have the Radeon Pro Vega II in slots 1 and 2, and was planning to put the Sonnets in slots 3 and 4.

2. Which RAID option would be ideal? RAID 4 or RAID 5?

3. Do I need to 'enable trim' on these drives? Is this still done in Terminal and can someone point to the correct commands for Catalina? (Don't have any real experience in doing this, apologies if I should be asking this elsewhere).

I can’t think of any reason to use RAID 4 on any new setup.

edit: it’s effectively deprecated and softraid probably doesn’t even support it.

Edit 2: although given what I’ve read about how long it’s been since softraid was last updated maybe they still do ?
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.