MP 1,1-5,1 Mac Pro 3,1 NVMe support + Upgrade Guide + Questions

[joevt]

I think what you had seen before was a EFI shell script being run for the patched APFS loader, and now that APFS is native in your bootROM that no longer needs to run.

Yes, it was the EFI Shell.

If your computer doesn't have APFS support, then the patcher installs the EFI Shell (named BootX64.efi which is the default name for the EFI startup program), apfs driver (named apfs.efi), and EFI startup shell script (named startup.nsh) into the EFI partition. The patcher sets the EFI Shell as the EFI startup program.

The first thing the EFI Shell does is show a list of mapped devices (hard drives, partitions, file systems) as a bunch of scrolling text. Then it launches the startup.nsh. The script loads the APFS driver, connects devices to the driver, adds new file system mappings for the apfs partitions, then it executes the boot.efi program on the macOS apfs partition.

You can install the EFI Shell with rEFInd to see how it works. Or you can create a virtual machine with UEFI bios in Parallels Desktop.app (or Virtual Box?) and play with the EFI Shell there. The map command shows the mapped devices.

 

[SheetLorde]

Hi, this guide helps a lot, thanks.
I installed Catalina on my SSD but Windows10 cannot boot on it, it seems the Windows bootloader treat NVME SSD
as external drive and fail to find system. Any solution? I tried disk copy and fresh install, both cannot find winload.exe

 

[joevt]

Booting from NVMe on MacPro3,1 can only be done using EFI mode.

Windows 10 booting on MacPro3,1 can only be done using BIOS mode which only supports booting from DVD or the four internal drive bays.

 

[SheetLorde]

Booting from NVMe on MacPro3,1 can only be done using EFI mode.

Windows 10 booting on MacPro3,1 can only be done using BIOS mode which only supports booting from DVD or the four internal drive bays.

I tried to use Windows installer in SATA bay to launch the one install on PCIE, still not working, cannot find winload.exe. It seems Windows believe the NVME disk is external disk

 

[tsialex]

I tried to use Windows installer in SATA bay to launch the one install on PCIE, still not working, cannot find winload.exe. It seems Windows believe the NVME disk is external disk

Look, it's very simple:

1. Windows to be installed on a NVMe drive, needs to be installed as an UEFI install. This can't be changed.
2. Mac Pro 3,1 only supports CSM/legacy Windows and don't accept an UEFI install. This can't be changed.

The moment that you understand the above two facts, you see that NVMe and Windows don't work with MP3,1.

The first Mac Pro that can install Windows to a NVMe drive is MP5,1, because Apple back in the day made the firmware in a way that Windows UEFI works, with several problems, but works.

 

[SheetLorde]

Booting from NVMe on MacPro3,1 can only be done using EFI mode.

Windows 10 booting on MacPro3,1 can only be done using BIOS mode which only supports booting from DVD or the four internal drive bays.

I tried to use Windows installer in SATA bay to launch the one install on PCIE, still not working, cannot find winload.exe. It seems Windows believe the NVME disk is external disk

 

[tsialex]

I tried to use Windows installer in SATA bay to launch the one install on PCIE, still not working, cannot find winload.exe. It seems Windows believe the NVME disk is external disk

Both @joevt and I explained that it's not possible.

Windows UEFI needs a computer that follows UEFI 2.3.1c spec, from June 2012. The last firmware update for MP3,1 is from March 2008. You can not make a CSM/legacy Mac boot Windows from a NVMe drive.

MP5,1 firmware was continuously updated by Apple until May 2019, that's how you can run UEFI Windows with MP5,1, with several problems, but you can. MP3,1 can't.

 

[joevt]

There are a couple options that might make it possible (I have tried neither, and they may have compatibility problems, and I will not discuss them further):

1) Use a new UEFI on top of Legacy/BIOS (maybe Clover installed as legacy might have a UEFI that can run Windows)?

2) Maybe some other legacy/BIOS booter can load an NVMe driver (I don't know if it exists). The people at neosmart.net may have options. There EasyBCD software has many legacy/boot options. Neogrub is a boot loader with many options. I think neogrub is related to grub4dos but I forget. Anyway, there may be NVMe/AHCI support for grub4dos. But it might only work for loading an image to RAM?

3) This might have info about #1 and #2:

NVMe Support for old Systems

Guides with and without BIOS Modding

www.win-raid.com

Either option is way too complicated and probably don't work. Just install Windows 10 to a 2.5" SSD connected to one of your four internal drive bays. Use the NVMe drive for data and apps.

 

[George Deley]

Hi, I tried flashing the boot ROM of my 3,1 last night and as far as I’m aware it didn’t do anything! (nVME boot support)

which in a way is a good/annoying thing!

bootrom version still says MP31.etc

not sure what happened but don’t really want to try it again, any suggestions.

couldnt find many simple guides on rEFind for nvme boot?

opencore bit confusing but seems simple. Tried it a week ago but just made my Mac slow. But figured outthat it may have overridden my MAXMEM command for the ram. So not sure weather to try again?

 

[triumphdelespai]

Hi there! i’ve already flashed the boot-room in my 3,1, even without having yet the nvme pcie adapter and the nvme drive. I guess that if i did all the steps and the mac started without any problem i’m out of risk zone brick my mac pro 3,1, am I?

Now i’m a bit confused to following the next steps on this thread and guide from handhelgames. Next steps should be apfs support and the pcietools thing. I'm on High Sierra with dosdude patcher.

I guess i have to do next with pcietools is giving the nvme pcie adapter the full speed it can get, right? Not sure because my second pcie free slot (x16) is already 2.0. So what’s the deal here? I know is also for booting with that pcie-e config on slot 2.

Another thing i have doubts is what kind of m.2 pcie nvme adapter should I buy, because i see that most of them are pcie 3.0. It needs to be pcie 2.0 compatible, isn’t it?


Thanks in advance

 

[joevt]

opencore bit confusing but seems simple. Tried it a week ago but just made my Mac slow. But figured outthat it may have overridden my MAXMEM command for the ram. So not sure weather to try again?

maxmem boot-arg is required for macOS if you have 64 GB of ram to speed things up. Similar maxmem setting is required for Windows 10 (a boot loader setting you can change with EasyBCD or other BCD editing app or command line). I'm not sure if there is a similar slow down due to 64 GB RAM in Ubuntu (I should run some disk benchmarks).
Anyway, OpenCore should be really good at setting boot-args.

I guess i have to do next with pcietools is giving the nvme pcie adapter the full speed it can get, right? Not sure because my second pcie free slot (x16) is already 2.0. So what’s the deal here? I know is also for booting with that pcie-e config on slot 2.

Most NVMe devices are PCIe 3.0. The MacPro3,1 has two PCIe 2.0 slots. The problem is that PCIe 3.0 devices might boot using PCIe 1.0 speed if they are in a PCIe 2.0 slot. Therefore you need a method to set it to PCIe 2.0 speed. I use a EFI Shell startup script for my PCIe 3.0 device to change it's speed to PCIe 2.0 (the startup script also loads the apfs driver and boots Catalina - it could probably load the NVMe driver too so no ROM patching would be required). Alternatives would be:
1) an EFI driver that can be loaded by rEFInd (it could be added to ROM like apfs and NVMe drivers)
2) a macOS startup script that uses pciutils
3) a macOS kext that doesn't require pciutils

The method to switch PCIe 1.0 speed to PCIe 2.0 speed is simple so any of the above should be easy to implement.

 

[triumphdelespai]

Most NVMe devices are PCIe 3.0. The MacPro3,1 has two PCIe 2.0 slots. The problem is that PCIe 3.0 devices might boot using PCIe 1.0 speed if they are in a PCIe 2.0 slot. Therefore you need a method to set it to PCIe 2.0 speed. I use a EFI Shell startup script for my PCIe 3.0 device to change it's speed to PCIe 2.0 (the startup script also loads the apfs driver and boots Catalina - it could probably load the NVMe driver too so no ROM patching would be required). Alternatives would be:
1) an EFI driver that can be loaded by rEFInd (it could be added to ROM like apfs and NVMe drivers)
2) a macOS startup script that uses pciutils
3) a macOS kext that doesn't require pciutils

The method to switch PCIe 1.0 speed to PCIe 2.0 speed is simple so any of the above should be easy to implement.

Now i understand why i need to setup the pcie to work with 3.0 devices. Thanks @joevt . I think i'll go for the 3 option via linkspd which seems to work fine.

 

[joevt]

linkspd

Do you mean the solution at #36 ? Or something else?

 

[triumphdelespai]

Do you mean the solution at #36 ? Or something else?

This thing: lnkspd

 

[triumphdelespai]

So I finally booted my mac pro 3,1 with nvme samsung evo 970 and getting 1425mb/s read and 1499 mb/s write. I have not done the apfs rom patcher thing because is working already fine on High Sierra formatted in HS+. For those of you who want to run slot 2 pcie 2.0 you can use lnkspd.pkg from github. worked fine for me.

Now i'm trying to figure out a solution for the drive to show it as internal device and not external.

 

[TJT49]

Hello everybody!

I'm newly owner of MP 3,1 and I want to discuss upgrading of this nice but old machine with you.

I suggest everyone to take part in creating this guide. Now I will post my achievement in making it bootable from NVMe SSD.

Bootable NVMe
Note: it can be dangerous for your MP 3,1, you can brick it if something goes wrong while flashing ROM. It's hard to repair, because you should unsolder chip, flash it with external usb programmer and then solder it back.


Useful links:
https://docs.google.com/document/d/1WNkM9LuGPq1sArO9EedWBHYq14NU7m-mDBLAWWJipyM/edit#
all credit for this instructional document goes to @handheldgames

There are three approaches to get NVMe booting. Two of them are without BootROM modification.

First approach without BootROM modification




Second approach without BootROM modification

You need install rEFInd and add NVMe driver. I will post step-by-step guide here a little later.


Approach with BootROM modification

I had a success with these steps which are almost the same as in Google Doc but you CAN use guide in Google Doc to flash your ROM. It's okey. ROMTool is GUI for flashrom tool I used in my guide. So these two guides are almost the same. Here is just another approach to do the same. My guide has steps to get nvme.ffs file with nvme driver. You need it because there is no link for this file in Google Doc now.
If you have old macOs which doesn't support ROMTool and have reasons not to update you Mac with newer OS but need NVMe drive you can use my steps.
Many thanks to @dosdude1 for this nice and easy to use GUI tool.

You should use guide in Google Doc and steps 1—3 in my guide , it would be much simpler and faster.
Many thanks to @handheldgames for the guide in Google Doc. I hope you will add steps 1—3 from my guide to your's one.

1. Download any version of MacOs Mojave with native NVMe boot support (look here). It can be 10.14.1 or any newer including latest for today (08/23/2019).
2. Right click on Install macOS Mojave, click Show Package Contents, go to Contents/Resources/Firmware,
copy MP51.fd to any folder.
3. Download binary UEFITool from here. Open MP5,1.fd in UEFITool, search text for NVMe, right click on NVMe branch, choose Extract as is..., save it with proper name like nvme.ffs
4. I had OsX 10.5 and no good internet connection on my MP3,1 (I can't download El Capitan for example, programs from Google Doc require higher version of MacOs than 10.5 I had on that moment) so I followed this steps. You can follow steps in Google Doc, but there were some reports here that ROMTool can be buggy. I created bootable USB stick with Grml Live Linux (you need 'full'). It's important to start MP3,1 in flash write mode. According to Google Doc power off MP3,1, wait at least 15 sec, press and hold power button, when LED above button starts to flash release power button. Then I booted my MP3,1 from USB stick (hold Option key when start and choose EFI).
In CLI write flashrom -p internal -r myrom.bin, after I mounted another USB stick, copied myrom.bin to it).
So now you have your ROM downloaded.
On another Mac I downloaded DXEInject, put myrom.bin and nvme.ffs in folder containing DXEInject and ran it in Terminal to create new patched ROM with NVMe support:
./DXEInject ./myrom.bin ./newromnvme.bin ./nvme.ffs
Now you have patched ROM and can flash it.
5. I copied newromvme.bin to MP3,1 via USB stick and ran flashrom:
flashrom -p internal -w newromnvme.bin
NOTE: You should be still in EFI FLASH MODE.
6.a. Then reboot and your MP3,1 is now NVMe bootable.
6.b. If something went wrong during startup, reset SMC and PRAM according to Google Doc. May be you bricked your MP3,1.


APFS Support

Thanks to @freqrider
freqrider's thread is here. See posts #18, #20, #25



Enable PCI 2.0 on slot 2.

1. Download pcietools from Google Doc
2. Follow this guide using notes in 2.a. to know your number for PCI slot: Post from joevt
2.a. Notes: after running sudo lspci -nnvt, find your NVMe drive in output and see pic in Spoiler how to edit script
In MP3,1 you should use 00:5 for Slot 1 and 00:1 for Slot 2

Spoiler

My result is 1050MB/s instead of 800MB/s.
When I did the same in Debian (with 64Gb RAM) I get 1280MB/s with hdparm -Tt /path-to-disk
With dd command after dropping buffer cache I got 1.4 GB/s. Is it possible to achieve this result or 1500MB/s on MacOS?

Questions about Slot 2 PCI 2.0

I use this PCIe to NVMe adapter.

According to Google Doc created by @handheldgames and other reports in Macrumors forum my SSD with simple x4 PCIe M.2 adapter should achieve 1500MB/s but it can't.

So I have 3 questions:
1. [SOLVED] How can I achieve speed more than 1050MB/s (1500MB/s) on MacOs?
I used Disk Sensei to check speed. It showed 1050MB/s. BlackMagick, AJA and dd CLI command shows 1500MB/s read and write.
2. [SOLVED] Did I have success with changing Slot 2 settings or something gone wrong and thats why NVMe SSD didn't achieve 1500MB/s reading speed?

3. [PARTIALLY SOLVED] Is it possible to make Slot 2 permanently PCI 2.0?



Max RAM in MP3,1. 62, 56 or 64Gb [Partially SOLVED]

As it was earlier reported, you can use only 56Gb in OsX 10.9+ without speed issues of your hard drives. In my case speed penalty was much more higher than 50% and macOS completely booted after 25 or more minutes

I have nice solution for those who run multiple OS (not only MacOS, but also Windows and/or Linux/Unix) to use 64Gb without speed penalty in MacOS.
Updated: you can run MacOs with 62Gb RAM available without speed penalty (with 64Gb installed)
You need simply limit max memory available for MacOS using this command:
sudo nvram boot-args="maxmem=57344"
So you will have 56Gb in MacOS and 64Gb in other OS.

Step-by-step guide:
1) Put RAM sticks in you MP3,1 to get total RAM lower or equal to 56Gb.
2) Run sudo nvram boot-args="maxmem=57344" in Terminal
3) Shutdown you MP3,1 and put 64Gb RAM.
4) Now you can use MacOS with 56Gb RAM available without any speed penalties and other OS with 64Gb RAM in total.
Note: 56Gb is 57344Mb

Can I have more RAM? Yes, you can run 62Gb without speed penalty!

Simply change 57344 to 63488. 62Gb is 63488Mb
sudo nvram boot-args="maxmem=63488" in Terminal

Now you will have 62Gb in MacOS and 64Gb in other OS.

Note: I had no speed penalties running 56Gb and 62Gb with 64Gb installed.
But my PCI to NVMe adapter is simple x4.
Probably you will have speed penalty if you are using x16 PCI to NVMe adapter. I have no such adapter so I can't test this approach with such adapter.
Please check and send your results.

See pics in Spoiler with my results.

Spoiler

Debian (and 99.99% all other Linux) can work with 64Gb in MP 3,1 without any speed penalty. (800MB/s in my case as in High Sierra without changing pci settings.)

I could run High Sierra with vm_compressor=1 on 56Gb without kernel panic, but with 64Gb SSD speed is very slow again.
Probably it's possible to port RAM management system from OsX 10.8 to use 64Gb without issues.

Curious notice: when I had 64Gb installed and one stick in Riser A in Slot 4 was damaged (red light on riser), total RAM was recognised as 56Gb in High Sierra and Debian. And About this Mac in memory tab showed that there were 2x 4Gb (!) sticks in Riser A Slots 3, 4. Also there was no speed penalty.


Using PCI Slot 2 with fat GPU

I don't have fat GPU in my MP3,1, but some guys told me that they can't access Slot 2 because of fat GPU.
I think in this case you can use flexible PCI riser like this (see pic in Spoiler)

Spoiler

Other

If you have much RAM you can disable SWAP using this command:
sudo nvram boot-args="vm_compressor=2"
Default value is 4.
If something gone wrong set default value or reset PRAM booting with holding Option+Cmd+P+R.


Useful Links

Mac Pro CPU Compatibility List
GPU Compatibility List for cMP
Blade SSDs - NVMe & AHCI
The Definitive Classic Mac Pro (2006-2012) Upgrade Guide


I invite @W1SS , @MrAverigeUser , @jimj740 , @freqrider , @tsialex , @dosdude1 , @joevt and others to discuss and complete this guide.
PS: If I mention users the way above will they receive notification about this thread?

PS2: Does anyone know how to contact with the authors of this Google Doc created by @handheldgames ?
Or where was it originally posted?

Colleague,

Thanks to you and so many who have and continue to compile this corpus of knowledge on this topic. I'm wondering, given the challenges associated with upgrading 3,1 machines, is it preferable or possible to swap out motherboards by scooping up a used 4,1 or 5,1 motherboard from eBay — to avoid the issues with the 3,1 logic board? I'm wondering if a 4,1 or 5,1 logic board will fit in a 3,1 case? My guess is the 4,1 and 5,1 boards present different external ports, and of course, different ROM's chips, etc.

Separately, if anyone can point me to a link for upgrade parts for the 3,1 machines. By parts, I mean SSD's, Chip's and PCIe upgrades that will work without altering the ROM or motherboards.

Many thanks. tom

 

[tsialex]

Colleague,

Thanks to you and so many who have and continue to compile this corpus of knowledge on this topic. I'm wondering, given the challenges associated with upgrading 3,1 machines, is it preferable or possible to swap out motherboards by scooping up a used 4,1 or 5,1 motherboard from eBay — to avoid the issues with the 3,1 logic board? I'm wondering if a 4,1 or 5,1 logic board will fit in a 3,1 case? My guess is the 4,1 and 5,1 boards present different external ports, and of course, different ROM's chips, etc.

Separately, if anyone can point me to a link for upgrade parts for the 3,1 machines. By parts, I mean SSD's, Chip's and PCIe upgrades that will work without altering the ROM or motherboards.

Many thanks. tom

MP4,1/MP5,1 backplanes and CPU trays can't be made to work with MP3,1 cases. Look at photos of a MP5,1 internally and you will easily see.

PCIe AHCI M.2 SSDs are supported natively by MP3,1 firmware.

PCIe SSDs - NVMe & AHCI

This is a general info thread for blade SSDs that can be used in the Mac Pro. This is a WikiPost so anyone with the proper credentials may edit it. NVMe SSDs can be used as a boot drive in the MP5,1 and MP6,1 with the latest firmware installed (beginning with version 140.0.0.0.0 for MP5,1 and...

forums.macrumors.com

 

[triumphdelespai]

So after some days of research and thanks the @dosdude1 , @handheldgames , @freqrider , @joevt , @mattc79 and @MisterAndrew for the hard work and the help, now i'm able to boot from my new samsung evo 970 on my mac pro 3,1 😄

 

[LightBulbFun]

1. Mac Pro 3,1 only supports CSM/legacy Windows and don't accept an UEFI install. This can't be changed.

so how many points is this worth then? 😄 (I sadly dont have a MacPro3,1 to test, but with some slight finagling I think I could boot windows in "U"EFI mode on a MP3,1)

(this was bare metal I did not use any 3rd party boot loaders or such)

 

[tsialex]

so how many points is this worth then? 😄 (I sadly dont have a MacPro3,1 to test, but with some slight finagling I think I could boot windows in "U"EFI mode on a MP3,1)

(this was bare metal I did not use any 3rd party boot loaders or such)

Nice accomplishment. Now please make it reproducible, document it and post a tutorial about how to install Windows UEFI on a MP2,1.

 

[LightBulbFun]

Nice accomplishment. Now please make it reproducible, document it and post a tutorial about how to install Windows UEFI on a MP2,1.

its actually pretty damn simple,

I just setup windows 10 in UEFI mode on another HDD (in the above case via VMware fusion VM and raw disk pass through IIRC)

then just dropped into the MP and hit EFI Boot in the boot picker

the MP firmware before MP5,1 (and most Macs from before Mid-late 2009) dont support some sort of display protocol that the windows Boot loader uses

so it LOOKS LIKE it freezes on the boot picker, but actually windows is still booting up, you just cant see it but if you have a compatible graphics card (I used an Apple OEM Radeon HD 5770)

then once the OS loads and fetches drivers for the GPU/when the GPU drivers kick in, the display then comes to life, and you can use windows

its also how I got windows on my Xserve3,1 (using an unflashed GeForce GT 740 and then a GT 640 I flashed latterly, and I pulled the onboard MXM GT 120 as Tesla based NVIDIA cards dont work in windows UEFI mode on any mac sadly they just cause a BSOD (you can get em to work but not without getting freaky in an EFI shell...)

making me the first person to run windows on an Xserve I think LOL

(sadly Windows seems to BSOD on something when I try it with my Xserve1,1)

as a side note as you could probably tell from above, I dont think windows supports "mixed mode" booting sadly, ie if the machine has a 32Bit EFI then you can only boot 32bit windows sadly

 

[Dayo]

Has anyone actually successfully installed UEFI Windows on a 3,1 apart from this 2,1 outlier with restrictions and unknowns by @LightBulbFun?

 

[DaddyMac502]

Hello ?
Anyone happen to have a tutorial
"Bootable NVMe without BootROM modification"

My name is DaddyMac502 registered user at macRumors.com (forums)
This is in reference to https://forums.macrumors.com/threads/mac-pro-3-1-nvme-support-upgrade-guide-questions.2194878/
Bootable NVMe

Second approach without BootROM modification
You need install rEFInd and add NVMe driver. I will post step-by-step guide here a little later.
Natively Mac Pro 3.1 does not support bootable NVMe drives! Currently I have a Mac Pro 3.1

 

[DaddyMac502]

So after some days of research and thanks the @dosdude1 , @handheldgames , @freqrider , @joevt , @mattc79 and @MisterAndrew for the hard work and the help, now i'm able to boot from my new samsung evo 970 on my mac pro 3,1 ?

Hello triumphdelespai
Would you happen to have a tutorial
"Bootable NVMe without BootROM modification"

My name is DaddyMac502 registered user at macRumors.com (forums)
This is in reference to https://forums.macrumors.com/threads/mac-pro-3-1-nvme-support-upgrade-guide-questions.2194878/
Bootable NVMe

Second approach without BootROM modification
You need install rEFInd and add NVMe driver. I will post step-by-step guide here a little later.
Natively Mac Pro 3.1 does not support bootable NVMe drives! Currently I have a Mac Pro 3.1

 

[triumphdelespai]

Hello triumphdelespai
Would you happen to have a tutorial
"Bootable NVMe without BootROM modification"

My name is DaddyMac502 registered user at macRumors.com (forums)
This is in reference to https://forums.macrumors.com/threads/mac-pro-3-1-nvme-support-upgrade-guide-questions.2194878/
Bootable NVMe

Second approach without BootROM modification
You need install rEFInd and add NVMe driver. I will post step-by-step guide here a little later.
Natively Mac Pro 3.1 does not support bootable NVMe drives! Currently I have a Mac Pro 3.1

You just need to follow this guide. I did it with boot rom modification and worked fine.

NVMe boot support how-to guide for the Mac Pro 3,1 / 4,1 / 5,1

The Mac Pro NVMe upgrade Apple forgot... An easy-to-follow step-by-step guide Supporting the Mac Pro 3,1 / 4,1 / 5,1 EFI The following document provides an easy to follow, step-by-step process for upgrading your Apple Mac Pro to include to support NVMe PCIe SSDs. Note: If you are running NVi...

docs.google.com