MP 1,1-5,1 5,1 won't boot from PCIe anymore

[JohnHa]

I've been running Debian 9 (Proxmox) on my 5,1 for the past couple years and it recently stopped booting from the PCIe drive. I have had it booting from an ssubx 1TB Apple (Samsung) drive pulled from a 2015 Macbook Pro in a cheap PCIe sled since November 2017. And I've also had an LSI SAS 9207-8i PCIe controller from December 2019, which has been working fine (non-boot).

But recently it just stopped being able to boot from any boot drive plugged into a PCIe slot. I get a blank grey screen which turns into a flashing grey ? folder. I have reset the PRAM and SMC several times. It boots fine from one of the four built-in sata2 bays. I currently have it booting from a 10k raptor in Bay2, but I'd like to be able to boot from a PCIe SSD again.

I don't have a Metal card (just the 5770 it came with), so I haven't updated to Mojave to update the firmware. I don't know which firmware it's running on... The last macOS that was installed on it was Sierra or perhaps High Sierra. But I don't run macOS on it, and I am not worried about Intel microcode vulnerabilities at the moment, I'm just trying to make it boot. I'll get around to buying a cheap Metal card or somehow try to borrow one later on, if ever.

If I have a bootable USB thumb drive, and I hold down the Option key upon boot, it shows me the PCIe SSD as a boot option, and I can select it and it boots fine. But if there is no thumb drive and I'm not holding down the Option key, then it fails to boot automatically from the PCIe boot drive (even if there are absolutely no other drives connected to it - internal or external).

I have removed all PCIe cards except for the 5770 and have tested the PCIe SSD in slots 2, 3 and 4 and it doesn't matter, it still doesn't boot reliably.

These configurations don't boot either (just giving the grey screen):
Bay2: the currently working bootable raptor drive plugged in
PCIe: either the SSD or the LSI card (no connected drives)
No other drives connected.

If I pull the PCIe card out, then it boots from Bay2 fine. Basically putting in one of my PCIe drive controller cards causes the Mac Pro not to boot.

I have a Sonnet Tempo SSD Pro Plus, and I tried to put that in (with an SSD attached) PCI slot 4 with no other drives except the bootable USB thumb drive, and again just a grey screen (holding Option key down).

I also have a generic USB 3.0 PCIe card, which I put into slot 2. I booted it with the raptor in Bay2, and it recognizes the thumb drive I put into the USB 3 card. So it has no problem booting with the USB 3 PCIe card in it.

I am just baffled... not sure which step to take next. If anyone has any clues, I would appreciate it! Thank you!

 

[startergo]

Did you try 4 consecutive NVRAM resets?

 

[JohnHa]

Great question - I did not. I only did CMD+Option+P+R for 3 chimes. I just did it for 5 chimes, and then when I let go of the keys, it chimed a 6th time before starting up. At first I get a grey screen, and then a couple interminable minutes later it shows the flashing grey ? folder icon.

If I start it while pressing the left Option key, I get a disk selection menu that shows one option, the PCIe SSD in slot 4 (as expected, as it's the only drive connected to the system). I just hit Enter and it boots the SSD perfectly.

But if I reboot or cold start it, I get the flashing grey ? folder. Nuts!

Any opinions from the wise ones here as to whether this is likely some sort of unfixable hardware failure, or if it's something software related that I can overcome? ?

p.s. booting proxmox on a 10k raptor in one of the stock bays runs surprisingly fast and it's usable. but of course I'm trying to get my much faster PCIe SSD back...

 

[joevt]

It boots fine from one of the four built-in sata2 bays. I currently have it booting from a 10k raptor in Bay2, but I'd like to be able to boot from a PCIe SSD again.

How about a SATA SSD in one of the drive bays?

If I have a bootable USB thumb drive, and I hold down the Option key upon boot, it shows me the PCIe SSD as a boot option, and I can select it and it boots fine. But if there is no thumb drive and I'm not holding down the Option key, then it fails to boot automatically from the PCIe boot drive (even if there are absolutely no other drives connected to it - internal or external).

How about no thumb drive and holding the option key?
Once you're in the Apple Startup Manager, move to the one you want to boot, hold the control key down so the up arrow turns into a circular arrow, then boot it. This will set the selected item as the default boot item.

I have a Sonnet Tempo SSD Pro Plus, and I tried to put that in (with an SSD attached) PCI slot 4 with no other drives except the bootable USB thumb drive, and again just a grey screen (holding Option key down).

Remove it.

I also have a generic USB 3.0 PCIe card, which I put into slot 2. I booted it with the raptor in Bay2, and it recognizes the thumb drive I put into the USB 3 card. So it has no problem booting with the USB 3 PCIe card in it.

USB cards are usually ok.

 

[JohnHa]

Once you're in the Apple Startup Manager, move to the one you want to boot, hold the control key down so the up arrow turns into a circular arrow, then boot it. This will set the selected item as the default boot item.

This is a great trick I didn't know before, thank you.


Current progress:
1. Removed all drives except for Bay1 (1TB stock WD drive that came with the 5,1 with Apple logo)
2. Created a bootable El Capitan thumb drive and installed El Cap to Bay1
3. Booted into El Cap and did the usual suggested system upgrades successfully.
4. Booted into El Cap and tried to upgrade to High Sierra. In the process it said firmware needed upgrading, so I did it and I got the unusual "boop" sound so I assume the firmware upgraded successfully. Progress!
4. After reboot, it successfully chose Bay1 and booted into El Cap. Yay!
5. Tried to upgrade to High Sierra again. Failed with message "macOS could not be installed on your computer - File system verify or repair failed. Quit the installer to restart your computer and try again. [Restart]" (screenshot below)
6. Rebooted a couple times but it boots into that screen every time, so I am in a bootloop now.
7. Back to square 1: fresh install El Cap from thumb drive to Bay1. This time ran Disk Utility on Bay1 and it says it's OK.
8. Won't install El Cap (see 2nd screenshot below)

Next Steps:
A. Make High Sierra bootable usb drive and try installing High Sierra on Bay1
B. Worth a shot: link
C. If that fails... remove all drives and put the PCIe SSD back in and use the Apply Startup Manager then hold the Control key down trick to select the SSD as the boot drive
D. ?


FROM STEP #5:

FROM STEP #8:

 

[m4v3r1ck]

Following

 

[JohnHa]

Followup:

Huzzah! Disconnect power for 60 seconds. Hold down power button for 10 seconds with power cable removed. Reset PRAM 4 times. Clean install from High Sierra bootable USB thumb drive worked. Although during install I got a weird message "This disk is locked." regarding my disk in Bay1. (Screenshot below) So I went to Disk Utility and wiped it, giving it a slightly different name.

Anyway, with just Bay1 and the USB installer connected, the machine automatically boots into High Sierra successfully. I checked System Information and it reports Boot ROM Version MP51.0089.B00 so the High Sierra firmware is confirmed.


Next steps:

Remove all drives. Install PCIe SSD. Boot with Option key. Select PCIe SSD and hold down Control key so up arrow turns into a circular arrow and boot it. See if it works OK. Test reboot and cold boot to see if it automatically boots from the PCIe SSD. Fingers crossed.


SCREENSHOT REGARDING WEIRD "This disk is locked." MESSAGE:

 

[JohnHa]

4 PRAM resets. Then boot holding down Option key with PCIe SSD installed and no other drives. Yields blinking grey ? folder icon. No joy.

Install Bay1 drive, leave PCIe SSD in place. Booting with Option key only shows two drives: Bay1 and the Recovery Volume. The PCIe SSD is not displayed as a recognized drive in the "Apple Startup Manager" screen. Bummer, it used to show there.

Never heard of this CMD+Option+O+F thing before, but I guess it's worth a shot: link

 

[JohnHa]

I give up. I am out of ideas. I have no idea why my Mac Pro 5,1 just decided it won't boot if any PCIe drive controllers are plugged in.

Findings:
1. Booted into High Sierra and updated all the system updates it wanted me to (took a long time on that old spinner, but it worked).
2. Disk Utility showed no other drives even though PCIe SSD was installed.
3. Many PRAM and SMC resets.
4. Holding Option key upon boot no longer shows the PCIe SSD as it used to, no matter how many times or PRAM resets or different PCI slots I tried.
5. CMD-Option+O+F did nothing for me.
6. Removing PCIe SSD and adding in LSI SAS card with 2 drives connected (non-boot) did not show up in Disk Utility (High Sierra)
7. Sonnet Tempo SSD Pro Plus in PCI slot 4 (which is a bootable card) was not recognized by the system.
8. Took many screenshots of the saga.

Current Situation:
A. System is booting fine from Bay2 with a 10k Raptor running Debian 11.
B. Bays 1, 3 and 4 have empty ext4 disks running fine.
C. Adding my PCIe SSD or an LSI PCIe disk controller card causes the machine to not boot. Holding Option key does not show the PCIe drives at all (this is different behavior from before the High Sierra [firmware] upgrade).
D. Adding the PCIe SSD and the LSI PCIe controller cards into another computer works fine.
E. The stock 5770 and a cheap USB 3.0 card are installed in PCI slots 1 and 2, respectively, and work fine.

 

[joevt]

Never heard of this CMD+Option+O+F thing before, but I guess it's worth a shot: link

That's insane. Command-Option-O-F is only for PowerMacs which have Open Firmware, not EFI.

locked? broken file systems? What is wrong with your disks? Wipe everything. Kill it with fire. Nuke it from orbit. It's the only way to be sure. https://knowyourmeme.com/memes/nuke-it-from-orbit

Maybe you have bad RAM. Mac Pros use ECC memory so it shouldn't be a problem? Or at least if there was, it would be detected.

Maybe your bootrom is bad. Get a ROM dump and verify.
https://forums.macrumors.com/threads/mp5-1-bootrom-thread-144-0-0-0-0.2132317/

But before all that, get a cheap Metal capable card and update the firmware with Mojave. You can go back to High Sierra and non-Metal card later. Just get the bootrom up to the latest so that you're working with a known base.

I don't use PCIe drive controllers because they might have weird PCI option roms that may affect legacy booting. NVMe drives should be ok. Your Mac Pro has 4 drive bays (6 including the ODD bays). Use them.

I think the issue with the Option key not showing PCIe SSD is because the current boot is set to internal drive bay. In that case only internal drive bay disks are shown in the Apple Startup Manager (boot picker - when you hold option key at boot). If you use the Startup Disk preferences panel to change the boot to a partition on the PCIe SSD (or the bless command if you want to change the boot to not macOS and not Windows), then maybe the Option key will show PCIe SSDs. If that's true (I remember a similar thing on my MacPro3,1 which is one reason why I don't use PCIe cards anymore), then a firmware update might fix it.

Or you could boot to a boot loader like RefindPlus. Press escape if it doesn't show all the disks to force a rescan.
Or OpenCore.

 

[JohnHa]

It's baffling.

All the disks have been nuked fresh, so no legacy things should be interfering. I will try to get a ROM dump, but wading through 186 pages on that thread sounds a bit daunting right now - I'm three straight days into trying to fix this issue.

Startup Disk preferences panel only shows the internal drives.

I once had 15 internal drives and 5 external (esata) drives running on this machine... 4 native bays, 4 in the ODD area, 4 in a DX4 sled, 2 in the Sonnet Tempo SSD Pro Plus, and the ssubx PCIe boot drive.

I am researching how to install RefindPlus right now. (OpenCore seems to require a Metal card and latest firmware.) Thank you for your help, @joevt ?

 

[joevt]

Startup Disk preferences panel only shows the internal drives.

Startup Disk preferences panel should show any disk that has macOS or Windows. Any other OS will require the bless command to set it as the default boot option.

Startup Manager when working properly should show any macOS, plus EFI partitions with a bootx64 in the default location, and HFS+ partitions with a blessed EFI file, and any MBR formatted disks for BIOS booting (which are called "Windows" even if they don't have Windows).

 

[Berenod]

Far as I know to be able to boot from PCIe NVME drives (what your ssubx 1TB Apple is) you need at least bootrom 140.0.0.0.0 but in post nr.7 you state that your system information reports MP51.0089.B00?

Bit of a mistery why it worked before, or somehow your bootrom has reverted to the older MP51 version, no idea how that would be possible though.

The main issue now being that you'd need a metal capable card to run the Mojave installer, which as a first step will upgrade to the current 144.0.0.0.0 bootrom. You don't actually have to install Mojave, just the bootrom update.

See if you can borrow such a card? Doesn't even have to be a Mac version (see here which cards would work: https://forums.macrumors.com/threads/gpu-compatibility-list-for-cmp.2174600/ )
Depending on what card you have now you might also need the mini PCIe 6 pin to regular PCIe 6pin cables.

Based on the comprehensive wiki regarding nvme/PCIe drive here:

PCIe SSDs - NVMe & AHCI

This is a general info thread for blade SSDs that can be used in the Mac Pro. This is a WikiPost so anyone with the proper credentials may edit it. NVMe SSDs can be used as a boot drive in the MP5,1 and MP6,1 with the latest firmware installed (beginning with version 140.0.0.0.0 for MP5,1 and...

forums.macrumors.com

 

[Dayo]

Basically putting in one of my PCIe drive controller cards causes the Mac Pro not to boot.

Sounds similar to an issue I had with my MP3,1. Ultimately realised it was a problem with the adaptor.

I wasn't primarily booting from the drive but the issue was that after working perfectly fine for a few years, it became the case that more often than not, the adaptor would simply not mount. I could always determine this was happening as there would be a 30 second delay in the boot process.

As said, I wasn't booting from this but it would have been an issue similar to yours if it had been my boot disk and not mounted.

I am researching how to install RefindPlus right now.

You can try the link in my sig. Not sure if it would directly make a difference but the logs might be helpful at least and you could initiate a rescan as was mentioned. I could see the timing gap in my logs when I had the issue.

I ultimately stopped using PCIe as it was just a normal SSD there as an extra disk for backup. So I connected the disk to the DVD Bay PATA Port and powered it from one of the hidden unused SATA ports on the MP3,1 MotherBoard.

 

[JohnHa]

I never want to boot macOS again on this machine, so I'm trying to configure rEFInd properly so that macOS doesn't stage a coup and replace rEFInd: https://www.rodsbooks.com/refind/bootcoup.html#osx

p.s. @Berenod, there was a time before NVMe support came to the 5,1 that you could get an Apple SSD out of a Macbook Pro and use it to boot from a simple PCIe sled.

 

[Dayo]

... so that macOS doesn't stage a coup

Mac OS does not really do Boot Coups. Certainly will not if you never boot into Mac OS again.

EDIT ... Should have been clearer:
Mac OS Boot Coups can only happen when booted in Mac OS (Updates as well as edits of the EFI partition).

The RefindPlus/OpenCore setup created by MyBootMgr is immune to any Mac OS Boot Coups when all Mac OS boots are done via OpenCore

 

[joevt]

Far as I know to be able to boot from PCIe NVME drives (what your ssubx 1TB Apple is) you need at least bootrom 140.0.0.0.0 but in post nr.7 you state that your system information reports MP51.0089.B00?

PCIe AHCI M.2 drives exist. I think the ssubx is AHCI? Also, the Sonnet Tempo SSD Pro Plus is a PCIe SATA controller.

I never want to boot macOS again on this machine, so I'm trying to configure rEFInd properly so that macOS doesn't stage a coup and replace rEFInd: https://www.rodsbooks.com/refind/bootcoup.html#osx

p.s. @Berenod, there was a time before NVMe support came to the 5,1 that you could get an Apple SSD out of a Macbook Pro and use it to boot from a simple PCIe sled.

macOS doesn't stage coup's like Linux and Windows do. If you don't use the bless command or Startup Disk preferences panel or the Startup Manager control key thing, then the default boot option won't change. I've got like 30 boot partitions on my MacPro3,1. You'll want to learn how to mount EFI partitions and examine/modify the boot variables. macOS might change the default boot when installing a new version, but that's the same with any OS. You just need to know how to change the default boot back.

The dumpallbootvars command in the following script can show the nvram boot variables:

macOS nvram boot variables, device properties, EFI device paths

macOS nvram boot variables, device properties, EFI device paths - gfxutil.sh

gist.github.com

There's a command efibootmgr -v in Linux to do similar.
Keep a copy of the variables handy so you can see how they should be or how they change. Keep a copy of the command that is needed to set the default boot to what you want.

 

[JohnHa]

Resolved thusly:
1. Installed debian10 to one of the four built-in bays and booted into that. And installed PCIe SSD in slot 4.
2. Installed rEFInd (seemed nicer than RefindPlus GUI f*ckery in macOS).
3. Deleted macOS entry in efibootmgr
4. Re-ordered boot order with efibootmgr
5. System seems to reliably boot into the PCIe SSD running debian11 (Proxmox).
6. Wiped the drives in the four built-in bays.
7. Haven't tried to add the LSI PCIe card and move the 4 drives from the built-in bays to the ODD area to get SATA3 speeds.

Thank you for everyone's help! Especially @joevt's generosity (as always) ?

 

[joevt]

RefindPlus GUI f*ckery in macOS

Which GUI are you referring to? MyBootMgr? I've never tried it. RefindPlus can be installed without a GUI. You can just replace the bootx64.efi of rEFInd with bootx64.efi of RefindPlus, and adjust the config.conf file to match whatever you modified in the refind.conf file. Each has a corresponding .conf-sample version which you can use to compare with the .conf version to see what settings have been modified.

I have rEFInd installed on a HFS+ partition (using the --ownhfs option of the rEFInd install script). Installing to an HFS+ partition allows the Startup Manager to show the name of the partition instead of just "EFI" (newer Macs don't have this problem). For EFI partitions, you can add a .VolumeIcon.icns file so that at least each EFI partition can have a unique icon at least. Anyway, the HFS+ installation is setup to look like a macOS installation (/mach_kernel, /System/Library/CoreServices/boot.efi & SystemVersion.plist so it can be blessed by the Startup Disk preferences panel (to bypass the need for the bless command). For HFS+, there is bless info stored in the HFS+ header of the partition. APFS volumes have similar bless info. The bless command can be used to view that. The boot.efi file ID is stored in the bless info. For EFI or FAT partitions, the boot file is at a specific location (/EFI/bootx64.efi) or the entire path is stored in the EFI boot variables (so you can boot EFI files that are not at /EFI/bootx64.efi).

The RefindPlus GitHub page has a list of the new/modified features. I guess there's not much reason to install RefindPlus unless you want to get a non-Mac GPU or use those new features. If you encounter a bug in rEFInd, there is a chance that it may be fixed in RefindPlus or at least the debug log might help find the bug. The debug log can only be written to a EFI or FAT partition since the EFI HFS driver only supports reading.

Let us know how your testing of the other non-Mac Pro drives goes.

 

[Dayo]

RefindPlus can be installed without a GUI. You can just replace the bootx64.efi of rEFInd with bootx64.efi of RefindPlus

Explained on the GitHub page: https://github.com/dakanji/RefindPlus#installation

The RefindPlus GitHub page has a list of the new/modified features. I guess there's not much reason to install RefindPlus unless you want to get a non-Mac GPU or use those new features.

Outlined on the GitHub page: https://github.com/dakanji/RefindPlus#additional-configurable-functionality

Most are Mac OS related and seems OP just wants Linux in which case not much difference. Only thing of note I suppose, is the "protect_nvram" feature if he ever installs Windows on his cMP. Otherwise, the new features may not be of much interest in a pure Linux setting. RefindPlus does generally default to avoid hammering the fragile NVRAM on a cMP. Some of the steps have since been adopted upstream in rEFInd as well.

If you encounter a bug in rEFInd, there is a chance that it may be fixed in RefindPlus.

Quite a lot of them have been fixed and bug reports on the rEFInd discussion page are immediately fixed in RefindPlus (the rEFInd developer has not been as responsive as he used to be). I suppose they are not total show stoppers and rEFind is perfectly fine if preferred ... especially if only running Linux.

 

[JohnHa]

Cool to know that RefindPlus is competent... but I'll never install macOS or Windows on this box (predicting that the 16" arm MBP will be the bees knees). And `rEFInd` is in the debian repositories, so `apt install refind` and having the default just work (never saw the .conf file) seemed nice. I only tested it for a few hours, but now my boot experience is: grey Mac screen with no icons or anything for a few seconds, then `GRUB`. I'm happy with that.

Let us know how your testing of the other non-Mac Pro drives goes.

I am tired. I may never get around to installing the LSI SAS card and moving the four 3.5" spinners to the ODD area to get SATA3 speeds. Trying to borrow a metal2 card that will allow me to install Mojave temporarily to get the new firmware, then installing NVMe PCIe cards (and perhaps removing all spinning drives entirely) seems like the next logical step.

I am giving this mac away to a friend and I am glad I've had ventilation filters installed on the front cover the whole time I've owned it, because the inside is whistle clean, and I just took the whole thing apart and blew it all out and wiped down all the fan blades and everything. It's in such a clean state, I love it. Thank you guys!

 

[joevt]

but now my boot experience is: grey Mac screen with no icons or anything for a few seconds, then `GRUB`. I'm happy with that.

What, did your NVRAM variables change again? Maybe it changed because you installed a new OS? Are you checking efibootmgr after each boot? Is your battery at least 3V (3.3V normally)? Get your bootrom checked by @tsialex
Is it safe to do Mojave rom update if your rom is already broken? I guess so, otherwise there wouldn't be a way to fix MacPro5,1 with bad nvram that haven't bricked yet. But Mojave rom update doesn't affect NVRAM area so it will need to be checked anyway?

https://forums.macrumors.com/thread...-bootrom-upgrade-instructions-thread.2142418/
https://forums.macrumors.com/threads/mp5-1-bootrom-thread-144-0-0-0-0.2132317/

 

[joevt]

spinners to the ODD area to get SATA3 speeds

Maybe you'll feel batter about SATA2 if you compare random reads/writes instead of sequential reads/writes (AmorphousDiskMark.app).

 

[JohnHa]

Cool, I don't run macOS on the thing so I can't use that app right now, but if I had to guess, sata2 holds up pretty decently for random and is maybe like half the speed for sequential. I *was* surprised at how usable the 10k raptor in the sata2 bay was... and an ssd in one of those bays is probably decent for a lot of things.

 

[JohnHa]

What, did your NVRAM variables change again? Maybe it changed because you installed a new OS? Are you checking efibootmgr after each boot? Is your battery at least 3V (3.3V normally)? Get your bootrom checked by @tsialex
Is it safe to do Mojave rom update if your rom is already broken? I guess so, otherwise there wouldn't be a way to fix MacPro5,1 with bad nvram that haven't bricked yet. But Mojave rom update doesn't affect NVRAM area so it will need to be checked anyway?

Interesting... hadn't considered any of these things. I have been checking `efibootmgr -v` after each new boot and it's been stable. I'm not sure how to check the battery voltage on linux... maybe one of those commands that spits out the entire system info details... I didn't know @tsialex did bootrom checking work... I don't think I'll be acquiring a metal2 GPU anytime soon, so as long as the thing holds up under this situation, I'll just "if it's not broke, don't fix it"...