MP 1,1-5,1 Mac Pro 3,1 NVMe support + Upgrade Guide + Questions

alexffff · Aug 23, 2019

Hello everybody!

I'm newly owner of MP 3,1 and I want to discuss upgrading of this nice but old machine with you.

I suggest everyone to take part in creating this guide. Now I will post my achievement in making it bootable from NVMe SSD.

Bootable NVMe

Note: it can be dangerous for your MP 3,1, you can brick it if something goes wrong while flashing ROM. It's hard to repair, because you should unsolder chip, flash it with external usb programmer and then solder it back.

Useful links:

https://docs.google.com/document/d/1WNkM9LuGPq1sArO9EedWBHYq14NU7m-mDBLAWWJipyM/edit#

all credit for this instructional document goes to @handheldgames

There are four approaches to get NVMe booting. Three of them are without BootROM modification.

A) EFI Shell with startup.nsh script

mattc79 said:
Without any Boot-Rom Modification loading NVME and APFS EFI drivers from a non nvme drive
a) I booted dosdude Mojave Patcher and installed macOs on the NVMe disk using APFS. When applying the dosdude patches I chose the APFS patch. The latter will install a efi bootloader (actually an EFI Shell) in the EFI Partition which will load the apfs driver via a the startup.nsh script.
b) Now I wasn't able to boot the NVMe disk, because the drive cannot be seen by the firmware. I copied the EFI Folder of the EFI Partition to the EFI Partition of a thumb drive in order to do the initial boot from there loading then nvme and apfs driver so the NVMe disk can be read and macOs can be booted.
c) So I extracted a NVMe EFI driver from Mojave's 10.14.5 MP5,1.fd firmware file using UEFITool by "extract body" on the NVMe DXE Driver's "PE32 image section" -> Saved to nvme.efi next to apfs.efi . I modified the startup.nsh so that it also loads the nvme.efi file similarly to how it loads the apfs.efi. You can also start experimenting by entering the EFI Shell (hit ESC very fast)... and use the various commands like 'load' 'drivers' 'connect -r' 'map -u' . And that all worked pretty nice for me! So I knew the nvme driver of MP51 cannot be bad for my machine.

B) EFI Boot loader (rEFInd, RefindPlus or OpenCore)

You need install rEFInd or RefindPlus or OpenCore and set to load an NVMe driver such as NVMExpressDxe.

NB: RefindPlus can supply both NVMe and APFS functionality by default when detected as required without BootROM modification or driver installation.

C) Driver#### and DriverOrder NVRAM variables

Mac Pro 3,1 NVMe support + Upgrade Guide + Questions

Adding NVMe and apfs support to MacPro3,1 without flashing ROM or using a EFI boot loader or Shell: You can place the EFI drivers on a EFI or FAT or HFS+ partition, then set Driver#### and DriverOrder NVRAM variables using the commands discussed here. In my example below, I chose a folder...

forums.macrumors.com

D) BootROM modification

I had a success with these steps which are almost the same as in Google Doc but you CAN use guide in Google Doc to flash your ROM. It's okey. ROMTool is GUI for flashrom tool I used in my guide. So these two guides are almost the same. Here is just another approach to do the same. My guide has steps to get nvme.ffs file with nvme driver. You need it because there is no link for this file in Google Doc now.

If you have old macOs which doesn't support ROMTool and have reasons not to update you Mac with newer OS but need NVMe drive you can use my steps.

Many thanks to @dosdude1 for this nice and easy to use GUI tool.

You should use guide in Google Doc and steps 1—3 in my guide , it would be much simpler and faster.

Many thanks to @handheldgames for the guide in Google Doc. I hope you will add steps 1—3 from my guide to your's one.

1. Download any version of MacOs Mojave with native NVMe boot support (look here). It can be 10.14.1 or any newer including latest for today (08/23/2019).

2. Right click on Install macOS Mojave, click Show Package Contents, go to Contents/Resources/Firmware,

copy MP51.fd to any folder.

3. Download binary UEFITool from here. Open MP5,1.fd in UEFITool, search text for NVMe, right click on NVMe branch, choose Extract as is..., save it with proper name like nvme.ffs

4. I had OsX 10.5 and no good internet connection on my MP3,1 (I can't download El Capitan for example, programs from Google Doc require higher version of MacOs than 10.5 I had on that moment) so I followed this steps. You can follow steps in Google Doc, but there were some reports here that ROMTool can be buggy. I created bootable USB stick with Grml Live Linux (you need 'full'). It's important to start MP3,1 in flash write mode. According to Google Doc power off MP3,1, wait at least 15 sec, press and hold power button, when LED above button starts to flash release power button. Then I booted my MP3,1 from USB stick (hold Option key when start and choose EFI).

In CLI write flashrom -p internal -r myrom.bin, after I mounted another USB stick, copied myrom.bin to it).

So now you have your ROM downloaded.

On another Mac I downloaded DXEInject, put myrom.bin and nvme.ffs in folder containing DXEInject and ran it in Terminal to create new patched ROM with NVMe support:

./DXEInject ./myrom.bin ./newromnvme.bin ./nvme.ffs

Now you have patched ROM and can flash it.

5. I copied newromvme.bin to MP3,1 via USB stick and ran flashrom:

flashrom -p internal -w newromnvme.bin

NOTE: You should be still in EFI FLASH MODE.

6.a. Then reboot and your MP3,1 is now NVMe bootable.

6.b. If something went wrong during startup, reset SMC and PRAM according to Google Doc. May be you bricked your MP3,1.

APFS Support

APFS support can be added using any of the above methods used to add Bootable NVMe.

freqrider said:
You can inject APFS and APFSJumpstart.ffs extracted from same, 5,1fd rom to your 3,1 rom and reflash it. Or you can use Dosdude1's APFS Flasher tool to do basically the same thing. http://dosdude1.com/apps/APFS ROM Patcher.zip (Only on Intel Sata chipset macs!) Password for zip is APFS.

Thanks to @freqrider

freqrider's thread is here. See posts #18, #20, #25

Alternatively, you can use either one or both of RefindPlus and OpenCore to provide APFS Support without flashing your ROM.

RefindPlus: Provides APFS capability by default when APFS support is not available
OpenCore: Activate the EnableJumpstart config key

If you use MyBootMgr to set RefindPlus andOpenCore up, it will configure both tools automatically.

Enable PCI 2.0 on slot 2

A) pciutils and bash script

1. Download pcietools from Google Doc

2. Follow this guide using notes in 2.a. to know your number for PCI slot: Post from joevt

2.a. Notes: after running sudo lspci -nnvt, find your NVMe drive in output and see pic in Spoiler how to edit script

In MP3,1 you should use 00:5 for Slot 1 and 00:1 for Slot 2

My result is 1050MB/s instead of 800MB/s.

When I did the same in Debian (with 64GB RAM) I get 1280MB/s with hdparm -Tt /path-to-disk

With dd command after dropping buffer cache I got 1.4 GB/s. Is it possible to achieve this result or 1500MB/s on MacOS?

B) EFI Shell with startup.nsh script

You can use the EFI Shell mm commands in the startup.nsh script to try to force PCIe 2.0 speed.

rEFInd / Discussion / General Discussion: all my Mac OSX versions do not appear

sourceforge.net

C) EFI driver

Create an EFI driver to do the work. Then you can have it loaded using any of the above methods used to add Bootable NVMe. FixPCIeLinkRate.efi is one such driver.

Questions about Slot 2 PCI 2.0

I use this PCIe to NVMe adapter.

According to Google Doc created by @handheldgames and other reports in Macrumors forum my SSD with simple x4 PCIe M.2 adapter should achieve 1500MB/s but it can't.

So I have 3 questions:

1. [SOLVED] How can I achieve speed more than 1050MB/s (1500MB/s) on MacOs?

I used Disk Sensei to check speed. It showed 1050MB/s. BlackMagick, AJA and dd CLI command shows 1500MB/s read and write.

2. [SOLVED] Did I have success with changing Slot 2 settings or something gone wrong and thats why NVMe SSD didn't achieve 1500MB/s reading speed?

freqrider said:
If you are getting over 750MB/s on slot 2, you have successfully switched your pci speeds with pciutil. Your slower speeds are probably due to your HD or adapter. I'm getting 1500 MB/s r/w with an 970 EVO and the Lycom DT-120 in my 3,1.

3. [PARTIALLY SOLVED] Is it possible to make Slot 2 permanently PCI 2.0?

joevt said:
Anything is possible. I don't think anyone's done the work to make it happen. It's been discussed elsewhere. There are many ways.
1) EFI driver
2) Login Item
3) launch daemon
4) kext
5) etc.

Max RAM in MP3,1. 62, 56 or 64GB [Partially SOLVED]

As it was earlier reported, you can use only 56GB in OsX 10.9+ without speed issues of your hard drives. In my case speed penalty was much more higher than 50% and macOS completely booted after 25 or more minutes

I have nice solution for those who run multiple OS (not only MacOS, but also Windows and/or Linux/Unix) to use 64GB without speed penalty in MacOS.

Updated: you can run MacOs with 62GB RAM available without speed penalty (with 64GB installed)

You need simply limit max memory available for MacOS using this command:

sudo nvram boot-args="maxmem=57344"

So you will have 56GB in MacOS and 64FB in other OS.

Step-by-step guide:

1) Put RAM sticks in you MP3,1 to get total RAM lower or equal to 56GB.

2) Run sudo nvram boot-args="maxmem=57344" in Terminal

3) Shutdown you MP3,1 and put 64GB RAM.

4) Now you can use MacOS with 56GB RAM available without any speed penalties and other OS with 64GB RAM in total.

Note: 56GB is 57344MB

Can I have more RAM? Yes, you can run 62GB without speed penalty!

Simply change 57344 to 63488. 62GB is 63488MB

sudo nvram boot-args="maxmem=63488" in Terminal

Now you will have 62GB in MacOS and 64GB in other OS.

Note: I had no speed penalties running 56GB and 62GB with 64GB installed.

But my PCI to NVMe adapter is simple x4.

Probably you will have speed penalty if you are using x16 PCI to NVMe adapter. I have no such adapter so I can't test this approach with such adapter.

Please check and send your results.

See pics in Spoiler with my results.

Leopard (10.5)

Leopard has a 32-bit only kernel. A kernel panic occurs with maxmem=63488 but maxmem=32768 works.

Note that boot-args can be placed in /Library/Preferences/SystemConfiguration/com.apple.Boot.plist instead of NVRAM for flags that you don't want to apply to other OSs.

Snow Leopard (10.6)

Snow Leopard defaults to 32-bit kernel. A kernel panic occurs with maxmem=63488. You can add arch=x86_64 to boot-args to force using the 64-bit kernel.

Windows

You can set an option in the BCD using EasyBCD or BCDEdit. In EasyBCD, it's called "Deduct xx MB from physical memory". Just deduct 2048MB. Use CPU-Z to run benchmarks for before and after changing the setting.

Advanced Settings

EasyBCD offers a number of more-advanced features for power users. Be careful when modifying these options, as they may result in degraded PC performance or a non-booting PC. Make sure to always ba…

neosmart.net

Linux

Debian (and 99.99% all other Linux) can work with 64GB in MP 3,1 without any speed penalty. (800MB/s in my case as in High Sierra without changing pci settings.)

I could run High Sierra with vm_compressor=1 on 56GB without kernel panic, but with 64GB SSD speed is very slow again.

Probably it's possible to port RAM management system from OsX 10.8 to use 64GB without issues.

Curious notice: when I had 64GB installed and one stick in Riser A in Slot 4 was damaged (red light on riser), total RAM was recognised as 56GB in High Sierra and Debian. And About this Mac in memory tab showed that there were 2x 4GB (!) sticks in Riser A Slots 3, 4. Also there was no speed penalty.

Using PCI Slot 2 with fat GPU

I don't have fat GPU in my MP3,1, but some guys told me that they can't access Slot 2 because of fat GPU.

I think in this case you can use flexible PCI riser like this (see pic in Spoiler)

Other

If you have much RAM you can disable SWAP using this command:

sudo nvram boot-args="vm_compressor=2"

Default value is 4.

If something gone wrong set default value or reset PRAM booting with holding Option+Cmd+P+R.

Useful Links

Mac Pro CPU Compatibility List

GPU Compatibility List for cMP

Blade SSDs - NVMe & AHCI

The Definitive Classic Mac Pro (2006-2012) Upgrade Guide

I invite @W1SS , @MrAverigeUser , @jimj740 , @freqrider , @tsialex , @dosdude1 , @joevt and others to discuss and complete this guide.
PS: If I mention users the way above will they receive notification about this thread?

PS2: Does anyone know how to contact with the authors of this Google Doc created by @handheldgames ?
Or where was it originally posted?

joevt · Aug 23, 2019

alexffff said:
In CLI write flashrom -p internal -r myrom.bin, after I mounted another USB stick, copied myrom.bin to it).
...
5. I copied newromvme.bin to MP3,1 via USB stick and ran flashrom:
flashrom -p internal -r newromnvme.bin

Are the two flashrom commands supposed to look the same?

alexffff said:
There is no information about adding APFS support, but it is possible too. Someone please add guide for it.
My Samsung PM961 gains 800MB/s now.

There's an apfs driver that's used for Clover. Maybe it can be injected like the NVMe driver.

alexffff said:
System reports tells that Slot 2 is x4 with 2.5GT/s link speed.

System report only shows the speed at startup, not the current speed.

alexffff said:
Speed is 1050MB/s.

2. Did I have success with changing Slot 2 settings or something gone wrong and thats why NVMe SSD didn't achieve 1500MB/s reading speed?

That is faster than PCIe 1.0 x4 so the command worked.

alexffff said:
According to Google Doc and other reports in Macrumors forum my SSD with simple x4 PCIe M.2 adapter should achieve 1500MB/s but it can't.

1. How can I achieve speed more than 1050MB/s (1500MB/s)?

Use a different NVMe. Maybe 950 Pro or 970 Pro?
A better solution is to use a card with a PCIe 3.0 bridge chip and 16 upstream lanes in slot 1 or slot 2. They are expensive but these cards can carry 2, 4, or 6 NVMe drives and allow the NVMe drives to operate at PCIe 3.0 speed.

alexffff said:
3. Is it possible to make Slot 2 permanently PCI 2.0?

Anything is possible. I don't think anyone's done the work to make it happen. It's been discussed elsewhere. There are many ways.
1) EFI driver
2) Login Item
3) launch daemon
4) kext
5) etc.

alexffff said:
2. What does Red light LED mean?

Read the "Power-On Self Test: RAM and Processor Verification" section of the Mac Pro (Early 2008) Service Source manual.

alexffff said:
PS: If I mention users the way above will they receive notification about this thread?

I didn't get an email message or a website alert.

alexffff · Aug 23, 2019

joevt said:
Are the two flashrom commands supposed to look the same?

corrected, thank you

joevt said:
Use a different NVMe. Maybe 950 Pro or 970 Pro?

there were some reports about pm961 too. it was with 1500mb/s speed

dosdude1 · Aug 23, 2019

You don't need to use Linux to flash your BootROM, just use my ROMTool utility in Mac OS (ZIP password is "rom"). It has no problems flashing the BootROM of a Mac Pro 3,1 that I'm aware of.

alexffff · Aug 23, 2019

dosdude1 said:
You don't need to use Linux to flash your BootROM, just use my ROMTool utility in Mac OS (ZIP password is "rom"). It has no problems flashing the BootROM of a Mac Pro 3,1 that I'm aware of.

Yes, but I had no internet connection to download new osx vesion (I had 10.5) to run GUI tools so I used Linux Live CD with flashrom). Somebody here told that ROMTool can be a little buggy (I dont know reasons for it), so if somebody scared he or she can use Linux to flash BootROM, probably it is more reliable

dosdude1 · Aug 23, 2019

alexffff said:
Yes, but I had no internet connection to download new osx vesion (I had 10.5) to run GUI tools so I used Linux Live CD with flashrom). Somebody here told that ROMTool can be a little buggy (I dont know reasons for it), so if somebody scared he or she can use Linux to flash BootROM, probably it is more reliable

ROMTool, which is based on flashrom, only has problems on machines with the nVidia MCP79 chipset. All other systems work just fine with ROMTool.

alexffff · Aug 23, 2019

I have just put my NVMe adapter to slot 1 to test speed of SSD and GPU to slot 2. Booting is very slow, progress bar became full in 2 or more minutes instead of 20-30 second. Now it's full for 5 minutes but I still can't see login window. Why? Finally in 10-15 minutes I got login windows, but it is very laggy. Still wondering why
upd: when I changed places of GPU and RAM adapter, I also changed one RAM stick. Now I have no red light and RAM recognised as 64Gb. So this is a reason for lags. But this lags are very terrible. It looks like ssd speed divided not by 2 as it was reported but by 20 or more. It took about 25 minutes to begin operate with macOS and it was very laggy.
Just removed 2 sticks from A riser from slots 3 and 4. Speed is now 800MB/s and 1050MB/s after updating pci speed. Still wondering how to achieve speed about 1500MB/s

alexffff · Aug 23, 2019

dosdude1 said:
ROMTool, which is based on flashrom, only has problems on machines with the nVidia MCP79 chipset. All other systems work just fine with ROMTool.

Added this information. Thank you

alexffff · Aug 23, 2019

Does anyone know how to contact with the authors of this Google Doc?
Or where was it originally posted?

crjackson2134 · Aug 23, 2019

alexffff said:
Does anyone know how to contact with the authors of this Google Doc?
Or where was it originally posted?

[SIZE=4][COLOR=#0000b3][B]@handheldgames[/B][/COLOR][/SIZE]

Ludacrisvp · Aug 23, 2019

I injected APFS into my 3,1 rom along with nvme ... it has been a long time since I did it so I don’t recall the exact steps taken off hand. But it works great.
I’ve never been able to get pcie speeds to change in mine though, I still haven’t added an nvme drive though but have various raid cards that don’t seem to be able to do this.

joevt · Aug 23, 2019

alexffff said:
there were some reports about pm961 too. it was with 1500mb/s speed

Different NVMe sizes or different NVMe controller firmwares can have different performance. Different Mac Pros? Different OS versions? Is that a read speed or a write speed?

For example, I can get better performance from a 512MB 950 Pro than I can get from a 1TB 960 Pro in some situations (Thunderbolt connection - I haven't tried PCIe recently).

There could be issues with the adapter (not likely). I have not studied this: is there info in the "lscpi -vvv" output that shows errors/retries with PCIe reads and writes?

Ludacrisvp said:
I’ve never been able to get pcie speeds to change in mine though, I still haven’t added an nvme drive though but have various raid cards that don’t seem to be able to do this.

Only PCIe 3.0 raid cards will require a speed change and only if they are in slot 1 or slot 2 of a MacPro3,1. If the raid cards are PCIe 2.0 you can use a speed change to test PCIe 1.0 speed.

Do you have a specific example? Check "sudo lspci -nnvvv" to make sure the higher speed is in the capabilities of the root port and the device. Apply the fast.sh commands to the correct root port.

alexffff · Aug 24, 2019

joevt said:
Different NVMe sizes or different NVMe controller firmwares can have different performance. Different Mac Pros? Different OS versions? Is that a read speed or a write speed?

My drive is 512Gb which is optimal to receive highest performance.
It's read speed ofc

joevt said:
There could be issues with the adapter (not likely). I have not studied this: is there info in the "lscpi -vvv" output that shows errors/retries with PCIe reads and writes?

Yes, there are some information there

Code:

-[0000:00]-+-00.0  Intel Corporation 5400 Chipset Memory Controller Hub [8086:4003]
           +-01.0-[01]----00.0  Advanced Micro Devices, Inc. [AMD/ATI] RV630 XT [Radeon HD 2600 XT] [1002:9588]
           +-05.0-[02]----00.0  Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]


02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (prog-if 02 [NVM Express])
    Subsystem: Samsung Electronics Co Ltd Device a801
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 256 bytes
    Interrupt: pin A routed to IRQ 19
    Region 0: Memory at 90a00000 (64-bit, non-prefetchable)
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable+ Count=1/32 Maskable- 64bit+
        Address: 00000000fee00000  Data: 4074
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <64us
            ClockPM+ Surprise- LLActRep- BwNot-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [b0] MSI-X: Enable- Count=33 Masked-
        Vector table: BAR=0 offset=00003000
        PBA: BAR=0 offset=00002000


00:05.0 PCI bridge [0604]: Intel Corporation 5400 Chipset PCI Express Port 5 [8086:4025] (rev 20) (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 256 bytes
    Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
    I/O behind bridge: 0000f000-00000fff
    Memory behind bridge: 90a00000-90afffff
    Prefetchable memory behind bridge: fffffffffff00000-00000000000fffff
    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
        PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    Capabilities: [50] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [58] MSI: Enable- Count=1/2 Maskable- 64bit-
        Address: fee00020  Data: 0000
    Capabilities: [6c] Express (v2) Root Port (Slot+), MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0
            ExtTag- RBE+
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #8, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
            ClockPM- Surprise+ LLActRep+ BwNot+
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
        SltCap:    AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
            Slot #5, PowerLimit 0.000W; Interlock- NoCompl-
        SltCtl:    Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
            Control: AttnInd Off, PwrInd Off, Power- Interlock-
        SltSta:    Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
            Changed: MRL- PresDet- LinkState-
        RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
        RootCap: CRSVisible-
        RootSta: PME ReqID 0000, PMEStatus- PMEPending-
        DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
        DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [b0] Subsystem: Intel Corporation Device [8086:8086]

There is something that looks like errors (RlxdOrd+, NoSnoop+)

And there is something interesting:
DevCap: MaxPayload 256 bytes
DevCtl: MaxPayload 128 bytes

Is it possible to change MaxPayload for device to make it as in device capabilities? I googled a lot about it but no luck

alexffff · Aug 24, 2019

alexffff said:
Curious notice: when I had 64Gb installed and one stick in Riser A in Slot 4 was damaged (red light on riser), total RAM was recognised as 56Gb in High Sierra and Debian. And About this Mac in memory tab showed that there were 2x 4Gb (!) sticks in Riser A Slots 3, 4. Also there was no speed penalty.

Probably it's possible to use this feature to make macOS recognise 8Gb sticks as 4Gb sticks in slots 3 and 4. It can be useful for everyone who runs not only macOS on MP3,1

alexffff · Aug 24, 2019

alexffff said:
Max RAM in MP3,1. 62, 56 or 64Gb [Partially SOLVED]

As it was earlier reported, you can use only 56Gb in OsX 10.9+ without speed issues of your hard drives. In my case speed penalty was much more higher than 50% and macOS completely booted after 25 or more minutes

I have nice solution for those who run multiple OS (not only MacOS, but also Windows and/or Linux/Unix) to use 64Gb without speed penalty in MacOS.
You need simply limit max memory available for MacOS using this command:
sudo nvram boot-args="maxmem=57344"

Step-by-step guide:
1) Put RAM sticks in you MP3,1 to get total RAM lower or equal to 56Gb.
2) Run sudo nvram boot-args="maxmem=57344" in Terminal
3) Shutdown you MP3,1 and put 64Gb RAM.
4) Now you can use MacOS with 56Gb RAM available without any speed penalties and other OS with 64Gb RAM in total.
Note: 56Gb is 57344Mb

Can I have more RAM? Yes, you can run 62Gb!

Simply change 57344 to 63488. 62Gb is 63488Mb
sudo nvram boot-args="maxmem=63488" in Terminal

Note: I had no speed penalties running 56Gb and 62Gb with 64Gb installed.
But my PCI to NVMe adapter is simple x4.
Probably you will have speed penalty if you are using x16 PCI to NVMe adapter. I have no such adapter so I can't test this approach with such adapter.
Please check and send your results.

See pics in Spoiler with my results.

Debian (and 99.99% all other Linux) can work with 64Gb in MP 3,1 without any speed penalty. (800MB/s in my case as in High Sierra without changing pci settings.)

I could run High Sierra with vm_compressor=1 on 56Gb without kernel panic, but with 64Gb SSD speed is very slow again.
Probably it's possible to port RAM management system from OsX 10.8 to use 64Gb without issues.

Curious notice: when I had 64Gb installed and one stick in Riser A in Slot 4 was damaged (red light on riser), total RAM was recognised as 56Gb in High Sierra and Debian. And About this Mac in memory tab showed that there were 2x 4Gb (!) sticks in Riser A Slots 3, 4. Also there was no speed penalty.

Added Solution to run MacOS (MP3,1) with 56Gb or 62Gb RAM available without speed penalty (with 64Gb (8x 8Gb) installed).
Please check it with more powerful NVMe adapter than PCI 4x.

freqrider · Aug 24, 2019

APFS Support

There is no information about adding APFS support here, but it is possible too. Someone please add guide for it.
-----------------------------------------------------------------------------------------------------------

You can inject APFS and APFSJumpstart.ffs extracted from same, 5,1fd rom to your 3,1 rom and reflash it. Or you can use Dosdude1's APFS Flasher tool to do basically the same thing. http://dosdude1.com/apps/APFS ROM Patcher.zip (Only on Intel Sata chipset macs!) Password for zip is APFS.

freqrider · Aug 24, 2019

If you are getting over 750MB/s on slot 2, you have successfully switched your pci speeds with pciutil. Your slower speeds are probably due to your HD or adapter. I'm getting 1500 MB/s r/w with an 970 EVO and the Lycom DT-120 in my 3,1.

alexffff · Aug 24, 2019

freqrider said:
APFS Support

There is no information about adding APFS support here, but it is possible too. Someone please add guide for it.
-----------------------------------------------------------------------------------------------------------

You can inject APFS and APFSJumpstart.ffs extracted from same, 5,1fd rom to your 3,1 rom and reflash it. Or you can use Dosdude1's APFS Flasher tool to do basically the same thing. http://dosdude1.com/apps/APFS ROM Patcher.zip (Only on Intel Sata chipset macs!) Password for zip is APFS.

thank you
Does APFS flasher tool contain APFSJumpstart?
What is APFSJumpstart? What is the difference from APFS.ffs?

freqrider said:
If you are getting over 750MB/s on slot 2, you have successfully switched your pci speeds with pciutil. Your slower speeds are probably due to your HD or adapter. I'm getting 1500 MB/s r/w with an 970 EVO and the Lycom DT-120 in my 3,1.

I used DiskSensei to check speed and I had 1050 MB/s after enabling PCI 2.0
BlackMagick shows 1500Mb/s read and 1000-1450 MB/s write speed.
Checking speed using dd command (with clear buffers) shows 1450 MB/s read

freqrider · Aug 24, 2019

alexffff said:
thank you
Does APFS flasher tool contain APFSJumpstart?
What is APFSJumpstart? What is the difference from APFS.ffs?

I used DiskSensei to check speed and I had 1050 MB/s after enabling PCI 2.0
BlackMagick shows 1500Mb/s read and 1000-1450 MB/s write speed.
Checking speed using dd command (with clear buffers) shows 1450 MB/s read

APFS Flasher contains both APFS.ffs and APFSJumpstart.ffs. I don’t know what the difference/functionality is, although I noticed the latest 5,1 bootrom does not contain APFS.ffs, only APFSJumpstart. (that I could find?) My modified 3,1 bootrom contains both, as well as the latest nvme.ffs

joevt · Aug 24, 2019

alexffff said:
Yes, there are some information there
There is something that looks like errors (RlxdOrd+, NoSnoop+)

And there is something interesting:
DevCap: MaxPayload 256 bytes
DevCtl: MaxPayload 128 bytes

Is it possible to change MaxPayload for device to make it as in device capabilities? I googled a lot about it but no luck

I've never tried. Download the "PCI Express® Base Specification Revision 3.0" and read about those registers.

alexffff said:
I used DiskSensei to check speed and I had 1050 MB/s after enabling PCI 2.0
BlackMagick shows 1500Mb/s read and 1000-1450 MB/s write speed.
Checking speed using dd command (with clear buffers) shows 1450 MB/s read

That's good. It means there's no problem here. You can also try "AJA System Test Lite.app" and "AmorphousDiskMark.app".

alexffff · Aug 24, 2019

joevt said:
I've never tried. Download the "PCI Express® Base Specification Revision 3.0" and read about those registers.

I will try, thank you!

joevt said:
That's good. It means there's no problem here. You can also try "AJA System Test Lite.app" and "AmorphousDiskMark.app".

Yep, AJA shows 1500MB/s too)

handheldgames · Aug 25, 2019

alexffff said:
Hello everybody!

I'm newly owner of MP 3,1 and I want to discuss upgrading of this nice but old machine with you.

I suggest everyone to take part in creating this guide. Now I will post my achievement in making it bootable from NVMe SSD.

Bootable NVMe
Note: it can be dangerous for your MP 3,1, you can brick it if something goes wrong while flashing ROM. It's hard to repair, because you should unsolder chip, flash it with external usb programmer and then solder it back.

Useful links:
https://docs.google.com/document/d/1WNkM9LuGPq1sArO9EedWBHYq14NU7m-mDBLAWWJipyM/edit#

There are three approaches to get NVMe booting. Two of them are without BootROM modification.

First approach without BootROM modification

Second approach without BootROM modification

You need install rEFInd and add NVMe driver. I will post step-by-step guide here a little later.

Approach with BootROM modification

I had a success with these steps which are almost the same as in Google Doc but you CAN use guide in Google Doc to flash your ROM. It's okey. ROMTool is GUI for flashrom tool I used in my guide. So these two guides are almost the same. Here is just another approach to do the same. My guide has steps to get nvme.ffs file with nvme driver. You need it because there is no link for this file in Google Doc now.
If you have old macOs which doesn't support ROMTool and have reasons not to update you Mac with newer OS but need NVMe drive you can use my steps.
Many thanks to @dosdude1 for this nice and easy to use GUI tool.

You should use guide in Google Doc and steps 1—3 in my guide , it would be much simpler and faster.
Many thanks to @handheldgames for the guide in Google Doc. I hope you will add steps 1—3 from my guide to your's one.

1. Download any version of MacOs Mojave with native NVMe boot support (look here). It can be 10.14.1 or any newer including latest for today (08/23/2019).
2. Right click on Install macOS Mojave, click Show Package Contents, go to Contents/Resources/Firmware,
copy MP51.fd to any folder.
3. Download binary UEFITool from here. Open MP5,1.fd in UEFITool, search text for NVMe, right click on NVMe branch, choose Extract as is..., save it with proper name like nvme.ffs
4. I had OsX 10.5 and no good internet connection on my MP3,1 (I can't download El Capitan for example, programs from Google Doc require higher version of MacOs than 10.5 I had on that moment) so I followed this steps. You can follow steps in Google Doc, but there were some reports here that ROMTool can be buggy. I created bootable USB stick with Grml Live Linux (you need 'full'). It's important to start MP3,1 in flash write mode. According to Google Doc power off MP3,1, wait at least 15 sec, press and hold power button, when LED above button starts to flash release power button. Then I booted my MP3,1 from USB stick (hold Option key when start and choose EFI).
In CLI write flashrom -p internal -r myrom.bin, after I mounted another USB stick, copied myrom.bin to it).
So now you have your ROM downloaded.
On another Mac I downloaded DXEInject, put myrom.bin and nvme.ffs in folder containing DXEInject and ran it in Terminal to create new patched ROM with NVMe support:
./DXEInject ./myrom.bin ./newromnvme.bin ./nvme.ffs
Now you have patched ROM and can flash it.
5. I copied newromvme.bin to MP3,1 via USB stick and ran flashrom:
flashrom -p internal -w newromnvme.bin
NOTE: You should be still in EFI FLASH MODE.
6.a. Then reboot and your MP3,1 is now NVMe bootable.
6.b. If something went wrong during startup, reset SMC and PRAM according to Google Doc. May be you bricked your MP3,1.

APFS Support

Thanks to @freqrider
freqrider's thread is here. See posts #18, #20, #25

Enable PCI 2.0 on slot 2.

1. Download pcietools from Google Doc
2. Follow this guide using notes in 2.a. to know your number for PCI slot: Post from joevt
2.a. Notes: after running sudo lspci -nnvt, find your NVMe drive in output and see pic in Spoiler how to edit script
In MP3,1 you should use 00:5 for Slot 1 and 00:1 for Slot 2

My result is 1050MB/s instead of 800MB/s.
When I did the same in Debian (with 64Gb RAM) I get 1280MB/s with hdparm -Tt /path-to-disk
With dd command after dropping buffer cache I got 1.4 GB/s. Is it possible to achieve this result or 1500MB/s on MacOS?

Questions about Slot 2 PCI 2.0

I use this PCIe to NVMe adapter.

According to Google Doc and other reports in Macrumors forum my SSD with simple x4 PCIe M.2 adapter should achieve 1500MB/s but it can't.

So I have 3 questions:
1. [SOLVED] How can I achieve speed more than 1050MB/s (1500MB/s) on MacOs?
I used Disk Sensei to check speed. It showed 1050MB/s. BlackMagick, AJA and dd CLI command shows 1500MB/s read and write.
2. [SOLVED] Did I have success with changing Slot 2 settings or something gone wrong and thats why NVMe SSD didn't achieve 1500MB/s reading speed?

3. [PARTIALLY SOLVED] Is it possible to make Slot 2 permanently PCI 2.0?

Max RAM in MP3,1. 62, 56 or 64Gb [Partially SOLVED]

As it was earlier reported, you can use only 56Gb in OsX 10.9+ without speed issues of your hard drives. In my case speed penalty was much more higher than 50% and macOS completely booted after 25 or more minutes

I have nice solution for those who run multiple OS (not only MacOS, but also Windows and/or Linux/Unix) to use 64Gb without speed penalty in MacOS.
Updated: you can run MacOs with 62Gb RAM available without speed penalty (with 64Gb installed)
You need simply limit max memory available for MacOS using this command:
sudo nvram boot-args="maxmem=57344"
So you will have 56Gb in MacOS and 64Gb in other OS.

Step-by-step guide:
1) Put RAM sticks in you MP3,1 to get total RAM lower or equal to 56Gb.
2) Run sudo nvram boot-args="maxmem=57344" in Terminal
3) Shutdown you MP3,1 and put 64Gb RAM.
4) Now you can use MacOS with 56Gb RAM available without any speed penalties and other OS with 64Gb RAM in total.
Note: 56Gb is 57344Mb

Can I have more RAM? Yes, you can run 62Gb without speed penalty!

Simply change 57344 to 63488. 62Gb is 63488Mb
sudo nvram boot-args="maxmem=63488" in Terminal

Now you will have 62Gb in MacOS and 64Gb in other OS.

Note: I had no speed penalties running 56Gb and 62Gb with 64Gb installed.
But my PCI to NVMe adapter is simple x4.
Probably you will have speed penalty if you are using x16 PCI to NVMe adapter. I have no such adapter so I can't test this approach with such adapter.
Please check and send your results.

See pics in Spoiler with my results.

Debian (and 99.99% all other Linux) can work with 64Gb in MP 3,1 without any speed penalty. (800MB/s in my case as in High Sierra without changing pci settings.)

I could run High Sierra with vm_compressor=1 on 56Gb without kernel panic, but with 64Gb SSD speed is very slow again.
Probably it's possible to port RAM management system from OsX 10.8 to use 64Gb without issues.

Curious notice: when I had 64Gb installed and one stick in Riser A in Slot 4 was damaged (red light on riser), total RAM was recognised as 56Gb in High Sierra and Debian. And About this Mac in memory tab showed that there were 2x 4Gb (!) sticks in Riser A Slots 3, 4. Also there was no speed penalty.

Using PCI Slot 2 with fat GPU

I don't have fat GPU in my MP3,1, but some guys told me that they can't access Slot 2 because of fat GPU.
I think in this case you can use flexible PCI riser like this (see pic in Spoiler)

Other

If you have much RAM you can disable SWAP using this command:
sudo nvram boot-args="vm_compressor=2"
Default value is 4.
If something gone wrong set default value or reset PRAM booting with holding Option+Cmd+P+R.

Useful Links

Mac Pro CPU Compatibility List
GPU Compatibility List for cMP
Blade SSDs - NVMe & AHCI
The Definitive Classic Mac Pro (2006-2012) Upgrade Guide

I invite @W1SS , @MrAverigeUser , @jimj740 , @freqrider , @tsialex , @dosdude1 , @joevt and others to discuss and complete this guide.
PS: If I mention users the way above will they receive notification about this thread?

PS2: Does anyone know how to contact with the authors of this Google Doc?
Or where was it originally posted?

Hey alexffff... I wrote the google doc that your are referring to. It's an original document with some of the info based on posts from MacRumors. The goal was pro take a difficult process and present it in a step-by-step format that could be leveraged as an upgrade guide for the masses. Good to know it still has some use.

joevt · Oct 19, 2019

alexffff said:
Added Solution to run MacOS (MP3,1) with 56Gb or 62Gb RAM available without speed penalty (with 64Gb (8x 8Gb) installed).
Please check it with more powerful NVMe adapter than PCI 4x.

I have a MacPro3,1, 3.2 Ghz eight core, no NVMe drives, just one HD and three SSDs connected to the four SATA drive bays. Catalina was running super slow with 64GB of ram. I used the "maxmem=63488" (62GB) nvram boot-arg, rebooted, and now everything is running normal speed again.

kjohansen · Oct 27, 2019

Where can I find info about upgrading my Mac Pro and Macbook Pro to at least 10.13

joevt · Oct 27, 2019

kjohansen said:
Where can I find info about upgrading my Mac Pro and Macbook Pro to at least 10.13

If your Mac is not supported by a version of Mac OS, then google "Install High Sierra on unsupported Macs". Do the same for Mojave and Catalina. You'll find a web page from DosDude1 and a corresponding MacRumors thread.

macOS High Sierra Patcher

macOS Mojave Patcher

macOS Catalina Patcher

macOS High Sierra (10.13) Unsupported Macs Thread

macOS High Sierra on Unsupported Macs For the first time since OS X Mountain Lion (10.8) in 2012, macOS Sierra (10.12) changed the official minimum requirements needed for installation, leaving many older (but still perfectly capable) Macs behind. Now macOS High Sierra (10.13) brings a new...

forums.macrumors.com

macOS 10.14 Mojave on Unsupported Macs Thread

macOS 10.14 Mojave on Unsupported Macs Machine Support Video Card Support Pre-Metal AMD video cards (Radeon HD 6xxx series and older without acceleration, Radeon HD 4xxx series and older with acceleration.) Pre-Metal Nvidia video cards (GeForce 5xx series and older, i.e...

forums.macrumors.com

macOS 10.15 Catalina on Unsupported Macs

How to update Catalina Machine Support Video Card Support Pre-Metal AMD video cards (Radeon HD 6xxx series and older without acceleration, Radeon HD 4xxx series and older with acceleration.) Pre-Metal Nvidia video cards (GeForce 5xx series and older without acceleration, GeForce 3xx...

forums.macrumors.com

MP 1,1-5,1 Mac Pro 3,1 NVMe support + Upgrade Guide + Questions

macrumors member

macrumors 604

macrumors member

macrumors 68030

macrumors member

macrumors 68030

macrumors member

macrumors member

macrumors member

macrumors 601

macrumors 6502a

macrumors 604

macrumors member

macrumors member

macrumors member

macrumors regular

macrumors regular

macrumors member

macrumors regular

macrumors 604

macrumors member

macrumors 68000

macrumors 604

macrumors regular

macrumors 604

Our Staff