MP 1,1-5,1 Mac OS 11.3 has broken support for older Mac Pros

[Syncretic]

This problem certainly is elusive. I have now gone through every single public source file that changed between 11.2 and 11.3, the entirety of both the kernel binaries, and the binaries (and, where possible, sources) of all the seemingly-relevant kexts, and I have yet to find anything that yields consistently good results. The one thing that seems to work reasonably well on my system (the gNVRAMSystemList patch) doesn't seem to do anything positive for anyone else. I identified a few other items that were promising (in that they were (1) clear changes between 11.2 and 11.3, and (2) probably represented a startup timing difference that could plausibly explain the symptoms we see), but patching those yielded only moderate (or no) gain. Interestingly, combining those with the gNVRAMSystemList patch actually made things worse on my system. I have rebooted so many times that I'm now planning to replace my BootROM chip, just to be safe.

My current thinking is this: the NVRAM is accessed through an EFI runtime service. Is it possible that supported machines have a newer EFI runtime that behaves just a little differently (especially for timing-sensitive operations)? The MP3,1 seems to use a M50FW016-class FWH flash, while the 4,1 uses an SST25VF032B-class SPI flash. The FWH interface is probably slower, but less time-sensitive than the SPI interface (although I'd guess both probably use the LPC controller for access), and that could account for why the MP3,1 seems to have a higher success rate than the MP4,1/5,1. If MacOS is expecting a fast result, or is sending commands/data too quickly, the older EFI interface (and/or the flash chip itself) could get confused, time out, or just hang.

I doubt they changed the actual EFI runtime interface, because if that were the case, I'd expect either 100% failures on older systems or runtime instability, and we don't see that.

So, given that the oldest supported systems are from 2013, my questions are:

• What flash chip(s) do the MP6,1/MBP11,* use?
• When did those systems last get a firmware update?

It may be time to disassemble the NVRAM EFI runtime and compare versions...

(I wrote the above before skimming the thread and discovering that Hackintoshes are also exhibiting problems. Do we have confirmation that Hacks are seeing the same problem, boot hangs at roughly the same place? Or are they seeing something different? If it's really the same, that puts a major damper on my EFI runtime hypothesis...)

(Regarding whether or not this is deliberate - I'd have to say it's not. If Apple wanted to make it nearly impossible for an old Mac Pro to run the new OS, it would be trivial for them to make it happen, and they haven't done that. I believe what we're seeing is simply indifference to the legacy systems - as others have pointed out, why bother testing the old hardware if you're not going to support it anyway?)

 

[tsialex]

This problem certainly is elusive. I have now gone through every single public source file that changed between 11.2 and 11.3, the entirety of both the kernel binaries, and the binaries (and, where possible, sources) of all the seemingly-relevant kexts, and I have yet to find anything that yields consistently good results. The one thing that seems to work reasonably well on my system (the gNVRAMSystemList patch) doesn't seem to do anything positive for anyone else. I identified a few other items that were promising (in that they were (1) clear changes between 11.2 and 11.3, and (2) probably represented a startup timing difference that could plausibly explain the symptoms we see), but patching those yielded only moderate (or no) gain. Interestingly, combining those with the gNVRAMSystemList patch actually made things worse on my system. I have rebooted so many times that I'm now planning to replace my BootROM chip, just to be safe.

My current thinking is this: the NVRAM is accessed through an EFI runtime service. Is it possible that supported machines have a newer EFI runtime that behaves just a little differently (especially for timing-sensitive operations)? The MP3,1 seems to use a M50FW016-class FWH flash, while the 4,1 uses an SST25VF032B-class SPI flash. The FWH interface is probably slower, but less time-sensitive than the SPI interface (although I'd guess both probably use the LPC controller for access), and that could account for why the MP3,1 seems to have a higher success rate than the MP4,1/5,1. If MacOS is expecting a fast result, or is sending commands/data too quickly, the older EFI interface (and/or the flash chip itself) could get confused, time out, or just hang.

I doubt they changed the actual EFI runtime interface, because if that were the case, I'd expect either 100% failures on older systems or runtime instability, and we don't see that.

So, given that the oldest supported systems are from 2013, my questions are:

• What flash chip(s) do the MP6,1/MBP11,* use?
• When did those systems last get a firmware update?

It may be time to disassemble the NVRAM EFI runtime and compare versions...

(I wrote the above before skimming the thread and discovering that Hackintoshes are also exhibiting problems. Do we have confirmation that Hacks are seeing the same problem, boot hangs at roughly the same place? Or are they seeing something different? If it's really the same, that puts a major damper on my EFI runtime hypothesis...)

(Regarding whether or not this is deliberate - I'd have to say it's not. If Apple wanted to make it nearly impossible for an old Mac Pro to run the new OS, it would be trivial for them to make it happen, and they haven't done that. I believe what we're seeing is simply indifference to the legacy systems - as others have pointed out, why bother testing the old hardware if you're not going to support it anyway?)

Almost all Macs from the 2012/2014 timeframe have the exactly same SPI flash memories from MXIC, just bigger sizes. When I reversed the SPI interface of MacPro5,1 I've started with a mid-2012 MacBook Pro schematic and it's almost identical, seems Apple just copied that whole part of the circuit for a lot of Macs.

I never read any reports of booting problems with now unsupported Macs like late-2012 Mac mini or mid-2012 MBPs that have the exact same SPI flash memories but double size.

The still supported late-2013 Mac Pro also have the exactly same one as used in mid-2012 Mac Pros, but the 64Mbit version instead of the 32Mbit one.

 

[tsialex]

@Syncretic
Apple used these SPI flashes with MacPro4,1 and MacPro5,1:

• with 2009 almost every backplane has SST25VF032B,
• with 2010 usually is MXIC MX25L3205D, sometimes can be MXIC MX25L3206E, very rarely is SST25VF032B,
• with 2012 usually is MXIC MX25L3206E, sometimes can be MXIC MX25L3205D.

Several 2012/2013/2014 Macs have Macronix MX25L6406E or Micron 25Q064A as SPI flash memories, but there are others supported and you can see the full table with efiflasher.

Below is the SPI flash from a 13" mid-2012 MacBook Pro:

 

[Syncretic]

I never read any reports of booting problems with now unsupported Macs like late-2012 Mac mini or mid-2012 MBPs that have the exact same SPI flash memories but double size.

Point taken, but the premise remains - if the systems that don't show these symptoms have different (from the MP4,1/5,1) NVRAM EFI runtime modules, that could plausibly explain what we're seeing (Hackintoshes aside - I'm hoping those don't actually show identical symptoms).

I think my next step, when I can find some time, is to disassemble and compare the NVRAM EFI runtime from my MP3,1 and MP4.5,1 to see how they differ. This hypothesis could be complete nonsense, but at this point, I'm running out of ideas... (One nice side-effect, if this hypothesis turns out to be valid, is that we should be able to just inject a working NVRAM EFI runtime module into the BootROM, not unlike adding APFS or NVMe support back in the day. Thatt's a big "if," though.)

 

[tsialex]

Point taken, but the premise remains - if the systems that don't show these symptoms have different (from the MP4,1/5,1) NVRAM EFI runtime modules, that could plausibly explain what we're seeing (Hackintoshes aside - I'm hoping those don't actually show identical symptoms).

I think my next step, when I can find some time, is to disassemble and compare the NVRAM EFI runtime from my MP3,1 and MP4.5,1 to see how they differ. This hypothesis could be complete nonsense, but at this point, I'm running out of ideas... (One nice side-effect, if this hypothesis turns out to be valid, is that we should be able to just inject a working NVRAM EFI runtime module into the BootROM, not unlike adding APFS or NVMe support back in the day. Thatt's a big "if," though.)

Won't be more productive to see if the runtime changed between MacPro5,1 to MacPro6,1? MacPro3,1 also have the same crashes, just not as much.

Btw, did you checked the source for any PCIe quirks removed between 11.2.3 and 11.3.1?

 

[khronokernel]

Do we have confirmation that Hacks are seeing the same problem, boot hangs at roughly the same place?

Same exact place for the hackintosh, and other Macs such as iMac8,1, iMac11,2, MacBook7,1 all experience these issues though less frequently. iMac8,1 being the worst for some reason

As tsialex mentioned, wouldn't spend much time diffing between the MacPro3,1 firmwares and 4/5,1 and would recommend looking to officially supported hardware such as MacPro6,1

 

[Bmju]

I’m not sure if it helps, but running OC 0.68 with graphical boot picker hidden, when I hold Escape and choose “Big Sur” 11.3.1 housed on its own PCIe NVMe, the machine boots properly seemingly every time. I’ll have to keep an eye on this and confirm.

Edit: added OC version and hidden boot picker

Is anyone else replicating or non-replicating this?

Also @RicheeThree it seems like the obvious thing - if this is making a difference on your setup - is that something (hardware?) is initialising while you are at the OC menu. In which case you would get exactly the same results if you don't hide your OC menu then show it with ESC, but rather just always show it (and either set timeout 0 making it always wait for you to press a key; or even, just set a few seconds timeout and let it start the OS for you automatically after that). Have you tried any of that, and if so with what results?

 

[Bmju]

NOTE: I'm not sure why, but OC was finicky about the format of this patch. If all of the elements between (and including) <dict> and </dict> weren't tabbed (i.e. no spaces to the left of each element, only tabs), it failed to apply the patch. This failure appears in the first blob of text that appears after selecting BS 11.3 (assuming you have verbose/debug boot-args enabled); you'll see a line containing OC: Prelinked patcher result: and (Patch gNVRAMSystemList (21may21)) - Not Found. If you see that, the patch didn't take, and you'll need to examine your config.plist to see what's what.

@Syncretic Completely separate to the excellent work you are doing in trying to track down this bug - tyvm! - I believe OC should not be sensitive to tabs, spaces, etc. as you are reporting, and I cannot replicate it. Could you provide a [code] section which includes a version of the patch with layout which does not work?

 

[Syncretic]

Same exact place for the hackintosh, and other Macs such as iMac8,1, iMac11,2, MacBook7,1 all experience these issues though less frequently. iMac8,1 being the worst for some reason

As tsialex mentioned, wouldn't spend much time diffing between the MacPro3,1 firmwares and 4/5,1 and would recommend looking to officially supported hardware such as MacPro6,1

If Hackintoshes are seeing the same thing, then the NVRAM angle is probably a dead-end. (BTW, the only reason I planned to compare 3,1 and 4/5,1 is that I have those systems in hand; if their NVRAM EFI modules were identical, I'd suspect the hypothesis was wrong, otherwise I'd start seeking a MP6,1 ROM to test. That's all moot now, of course.)

---

I don't have much time this weekend, but a thought occurred to me last night as I fell asleep, and maybe someone can look at it (or perhaps already has): The APFS format has been evolving. It changed a little between Catalina and Big Sur, and changed between dot-revs of Big Sur. On my system, both Catalina and Mojave can read and write an 11.1 APFS volume, but report Incompatible Disk for the 11.3 volume. I'm wondering if the 11.3+ disk format itself is part of the equation. Presumably, all of the glitchy installations (including Hackintosh) are booting from 11.3-formatted disks.

Has anyone tried pre-formatting the target disk to APFS using 11.1 or earlier prior to running the 11.3+ installer? Or does the 11.3+ installer forcibly reformat the drive anyway? Is there a way to restore an 11.3 installation onto a disk formatted by a pre-11.3 version? Conversely, is anyone who upgraded to 11.3+ from 11.2 or earlier via OTA experiencing these same symptoms? Does an OTA upgrade modify the disk format (i.e. is an OTA-upgraded-to-11.3 disk readable/writable from Catalina/Mojave)?

(Probably grasping at straws here, but that seems to be where I'm at...)

@Syncretic Completely separate to the excellent work you are doing in trying to track down this bug - tyvm! - I believe OC should not be sensitive to tabs, spaces, etc. as you are reporting, and I cannot replicate it. Could you provide a [code] section which includes a version of the patch with layout which does not work?

I thought it was odd as well; most parsers are free-form and ignore whitespace. If I recall correctly, the cut & paste I did substituted spaces for some or all of the tabs on the (Base64) data line only (all other lines were tabbed), and the patch failed until I manually changed the spaces to tabs. When I get some time I'll try to replicate it again (my config.plist has undergone a great many changes since then) and provide more details. Probably not until next week, though.

EDIT:

Btw, did you checked the source for any PCIe quirks removed between 11.2.3 and 11.3.1?

I did look at IOPCIFamily, IOUSBHostFamily/AppleUSB*HCIPCI, and AppleSMBusPCI, but maybe I'll give those more scrutiny in case I missed something - maybe also some ancillaries, such as IOBluetoothHostControllerPCIeTransport, AppleEmbedded*, etc. (I was initally trying to stick to the most common/central kexts).

 

[startergo]

Is anyone else replicating or non-replicating this?

I tried holding both escape and eject after the OC boot screen. Did not seem to make a difference.

 

[startergo]

Noticed that Catalina pauses much more frequently during the verbose boot process as compared to BS 11.4/5 and as a whole the booting process is much slower.

 

[JohnD]

Has anyone tried pre-formatting the target disk to APFS using 11.1 or earlier prior to running the 11.3+ installer? Or does the 11.3+ installer forcibly reformat the drive anyway?

I have Mojave, 11.3.1, 11.4RC, and 11.5b1 installed on my test machine. One of those installs (I believe the 11.3.1 HDD, easily booted and checked) started as a CCC clone of my primary system running 11.2.3, then upgraded to 11.3x. How do you check the version of APFS on each drive? I can boot it and check all volumes.

Assuming APFS version is not upgraded, there are 2 variables at play. 1) I had much better results booting from that 11.3x installation that was upgraded from 11.2.3, but still had the same behavior otherwise, and 2) It's an HDD, which we assumed was the reason for the much better success rate.

Another interesting point, now that I think about it, is that I made a CCC clone of that upgraded 11.3.1 HDD, to another HDD, when asked to test FileVault. I had worse results after FileVault encryption than the 11.3.1 HDD, but equal to the 11.4/11.5 installs. Does FileVault upgrade the APFS version upon encryption?

Let me know how to check APFS version and I'll gather all that data. Also, I can blow away any of those installs, perform a fresh install of any OS, and upgrade to any version of Big Sur.

 

[Macschrauber]

My 11.5b1 lives on an Apple AHCI Blade and I have a very good success rate. Its PCI2.0x4 (1000MB plus I/O). I once did a clone with ccc and startet with 11.2

there was a spinner and a sata ssd in the copy history.

so I guess its not the general slowness of a hdd what makes better success rates.

I have no wifi board in that mac if thats another data point.

regarding the firmware discussion: remember I flashed a original ancient 2010 Firmware (apfs via RefindPlus) and got exactly the same boot problems as with 144.0.0.0

 

[cdf]

My 11.5b1 lives on an Apple AHCI Blade and I have a very good success rate. Its PCI2.0x4 (1000MB plus I/O). I once did a clone with ccc and startet with 11.2

there was a spinner and a sata ssd in the copy history.

so I guess its not the general slowness of a hdd what makes better success rates.

I have no wifi board in that mac if thats another data point.

If you're referring to the SSUBX, I've tested that drive, and my success rate remains abysmal. However, my Mac Pro has a thunderbolt add-in card and a WiFi card.

 

[Dayo]

I flashed a original ancient 2010 Firmware (apfs via RefindPlus) and got exactly the same boot problems as with 144.0.0.0

RefindPlus would use APFS from the operating system being loaded. Not related to the firmware.

So it will use Mojave's APFs when loading Mojave and BS 11.3's APFS when loading BS 11.3 regardless of whether you have 144.0.0.0, the 2010, cMP 3,1 or any other firmware active.

EDIT: It would only do this for firmware without APFS support as 144.0.0.0 or any other firmware with APFS support would have enabled APFS (following the same principle) before RP is loaded.

 

[Macschrauber]

If you're referring to the SSUBX, I've tested that drive, and my success rate remains abysmal. However, my Mac Pro has a thunderbolt add-in card and a WiFi card.

Yes, exactly that blade. Just wanted to set the data point that access speed of the system disk is not the main problem.

 

[eVasilis]

Hi guys,
if the problem is found to be with the efi chip, would installing a mat card be any different?

 

[tsialex]

Hi guys,
if the problem is found to be with the efi chip, would installing a mat card be any different?

No, won't change anything. Several people here are using MATT cards as the main SPI flash, including me.

MATT card is for all effects, a clone of the Apple SPI circuit design using the exact same MXIC MX25L3206E flash memory of mid-2012 backplanes, just external/removable.

MATT cards are not a perfect solution, it's a very costly solution. It's most useful for developers or for people that don't want to have the trouble of repairing/replacing the backplane and don't mind the cost.

Edit:

I was looking at a photo of my MATT card and I noticed that my card have the usual mid-2010 MXIC MX25L3205D and not the more recent MX25L3206E. Other MATT cards owners that have newer made cards have the later SPI flash model. I've totally forgot about that.

 

[eVasilis]

just external/removable.

All the ones I found searching, plug onto the backplane. I cannot find any external/removable ones. Could you please point me to the right direction?

 

[tsialex]

All the ones I found searching, plug onto the backplane. I cannot find any external/removable ones. Could you please point me to the right direction?

Seems you didn't understood my point at all. The board connects to LITTLEFRANK connector and it's a external SPI flash and removable - if your backplane still has a working BootROM, if not you need the MATT card to be always connected.

 

[eVasilis]

Seems you didn't understood my point at all. The board connects to LITTLEFRANK connector and it's a external SPI flash and removable - if your backplane still has a working BootROM, if not you need the MATT card to be always connected.

Got it, thanks!

 

[borp99]

No, won't change anything. Several people here are using MATT cards as the main SPI flash, including me.

MATT card is for all effects, a clone of the Apple SPI circuit design using the exact same MXIC MX25L3206E flash memory of mid-2012 backplanes, just external/removable.

Discussing these MATT cards is all new(s) to me. I didn’t know until now that there was a LITTLEFRANK connector on the Mac Pro 5,1 backplane. Is the advantage, of buying and adding one, that you are writing NVRAM variables to a fresh flash chip, rather than the ageing original one?

[all of this testing/multiple-booting of BS can’t be helping in trying to reliably/stably move beyond 11.2.3]

I could only find a company called http://www.cmizapper.com/ selling these cards. Is that the only source/alternative? They say you need an EasyFlash adapter A4056 as well, but these aren’t listed on their site. Are there instructions or further info available elsewhere for any of us to ‘join the dots’? Is all of this collecting of hardware and programming beyond even a moderately computer-savvy punter?

PS. If all of this is possible, beyond a few ultra-tech members, is using the 2012 MATT card on a 2010 5,1 possible or advantageous? The cmizapper site says not to mix the Mac model’s cards.

 

[borp99]

It's the same model from early-2009 to mid-2012, it's just one MATT card for all years.

Be aware that most of what cmizapper writes, is plain wrong or not applicable at all to Mac Pros. They made a product with one intention in mind while we use it for a slightly different one and in a completely different way.

If you buy one, you will discard anything that cmizapper flashed to it. Read what I've written in the past, you will understand.

Thank you - I appreciate the quick response and direction to read further. Old news is still new news if you haven’t heard or seen it before.

I recently saved a good copy of my BootROM - after 6 chimes / had an empty (full free space) 2nd VSS Store and 40698 (bytes?) free of 65464 of the 1st VSS Store. However, it ‘could’ fail at an stage I guess.

 

[tsialex]

Discussing these MATT cards is all new(s) to me. I didn’t know until now that there was a LITTLEFRANK connector on the Mac Pro 5,1 backplane. Is the advantage, of buying and adding one, that you are writing NVRAM variables to a fresh flash chip, rather than the ageing original one?

[all of this testing/multiple-booting of BS can’t be helping in trying to reliably/stably move beyond 11.2.3]

I could only find a company called http://www.cmizapper.com/ selling these cards. Is that the only source/alternative? They say you need an EasyFlash adapter A4056 as well, but these aren’t listed on their site. Are there instructions or further info available elsewhere for any of us to ‘join the dots’? Is all of this collecting of hardware and programming beyond even a moderately computer-savvy punter?

PS. If all of this is possible, beyond a few ultra-tech members, is using the 2012 MATT card on a 2010 5,1 possible or advantageous? The cmizapper site says not to mix the Mac model’s cards.

I'm glad to answer, but this is a very old topic that I've written extensively about all pros and cons, use the search for MATT written by me, you will find more than you ever want to read about this.

SPI flash has a limited life, if your Mac Pro has being in use continuously, it's probably near the end of the useful life. I've explained this in detail before, including the common misconceptions about the tech involved.

Be aware that most of what cmizapper writes, is plain wrong/not applicable at all to Mac Pros. They made a product with one intention in mind while we use it for a slightly different one, in a completely different way, and MATT cards are not a SPI replacement solution cost effective for anyone that can solder a SOIC/SOJ. It's most useful for developers and for people that don't want to have the trouble of repairing/replacing the backplane and don't mind the cost.

You only need a MATT card, nothing else, and it's the same Chipmunk A4054 MATT card model from early-2009 to mid-2012. It's just one MATT card for all years. If you buy one, you will discard anything that cmizapper flashed to it since from factory all MATT cards have the same extremely dirty BootROM image, see the report below. All cards are clones of the GJ0340TXH2N Mac Pro that have a perfect example of a NVRAM volume with failed garbage collection until you flash your own clean BootROM dump.

If you don't have a clean BootROM image for your Mac Pro, you'll need to pay for a firmware reconstruction. Again, you won't use the BootROM image flashed to the MATT card, you will use your own Mac Pro dump.

I was looking at the photo of my MATT card below that shows the SPI and I noticed that my card have the usual mid-2010 MXIC MX25L3205D and not the more recent MX25L3206E. I've totally forgot that my card had the earlier one.

Other MATT cards owners that have newer made cards received it with the later SPI flash model.

 

[sfalatko]

I am one of the lurkers who is watching as the team works to figure out the issues in booting 11.3+ on our cMPs. I saw this today and wanted to point it out - Apple is definitely working on the boot process in macOS 11 - in case it provides any additional insight in helping figure out if we can get 11.3+ booting reliably on our MacPro5,1s.

M1 Macs running Big Sur 11.4 support external disks fully

macOS 11.4 has finally fixed all the problems with installing current and older macOS on external disks, and booting an M1 Mac from them.

eclecticlight.co

Regards,
Sfalatko