Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Status
The first post of this thread is a WikiPost and can be edited by anyone with the appropiate permissions. Your edits will be public.

skodises

Contributor
Jan 9, 2021
41
46
Just another testimonial for the SPI flash boot ROM reconstruction service provided by @tsialex, on my mid2010 5,1 with ancient boot room code. I also bought a MATT card, primarily for convenience, but also to mitigate any risks - because I could not afford to have my daily driver down for long. I followed his instructions to the letter, installed the MATT card, booted into a generic clean High Sierra install on the disk in Slot 1, disabled SIP, rebooted into firmware update mode, entered ROMtool, flashed the reconstructed image Alex provided, rebooted, and the machine came right up cleanly in my dosdude1-patched NVMe Catalina setup- boot chime and all. Total time: 15 minutes on a Saturday morning.

The thing that took the longest was booting into the clean HS install on that ancient spinner in Slot 1. It was as straightforward as it possibly could be, and I’m now much better protected against SPI flash death. Fantastic, and I’m very pleased with the result. It boots slightly faster, the chime is back for the first time in years, and warm-switching boot volumes works again. Worth every penny, and highly recommended!
 
Last edited:

trifero

macrumors 68030
May 21, 2009
2,956
2,796
Another succesfull scheduled ROM flash. Thanks again, @tsialex What a nice job.
 

Attachments

  • Captura de pantalla 2022-01-09 a las 6.50.23.png
    Captura de pantalla 2022-01-09 a las 6.50.23.png
    161.4 KB · Views: 191

cMP 51 neewbie

macrumors newbie
Jan 9, 2022
18
4
Oh no. It does not look good for me. According to a post in the OC Thread I dumped my BootROM and looked at the free space. According to @tsialex there has to be a minimum of 22000 of free space. Well, I am at 24000. Anything I can do about that/can someone please help me with that? I am still a student and need my machine daily for hours...So I can't face any troubles right now...
Edit: It's a mid 2010 dual CPU model. Currently running macOS 12.1 and Windows.
another Edit: the tool from MacSchrauber did NOT work for me. I also had to choose a different chip than the one that was recommended by the dumping tool.
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
Oh no. It does not look good for me. According to a post in the OC Thread I dumped my BootROM and looked at the free space. According to @tsialex there has to be a minimum of 22000 of free space. Well, I am at 24000. Anything I can do about that/can someone please help me with that? I am still a student and need my machine daily for hours...
First thing, it's an early-2009 or mid-2010/mid-2012? I suppose that is a dual CPU tray, correct?

Second, do a deep NVRAM reset (use a wired keyboard that works for NVRAM resets and keep pressed CMD-ALT-P-R until you heard the 5th chime) and dump/check the free space available again. If you don't get a drastic improvement, your garbage collection failed and you will need a clean-up/upgrade/reconstruction service of your BootROM image.
 

cMP 51 neewbie

macrumors newbie
Jan 9, 2022
18
4
First thing, it's an early-2009 or mid-2010/mid-2012? I suppose that is a dual CPU tray, correct?

Second, do a deep NVRAM reset (use a wired keyboard that works for NVRAM resets and keep pressed CMD-ALT-P-R until you heard the 5th chime) and dump/check the free space available again. If you don't get a drastic improvement, your garbage collection failed and you will need a clean-up/upgrade/reconstruction service of your BootROM image.
Thanks for the fast reply. I did a deep NVRAM rest and dumped afterwards. Unfortunately no comparison possible. Yes, it's a mid 2010 dual tray model.
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
Thanks for the fast reply. I did a deep NVRAM rest and dumped afterwards. Unfortunately no comparison possible. Yes, it's a mid 2010 dual tray model.
It's too low available space after a forced garbage collection, something is wrong. How many DIMMs/type/size you have installed?
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
It's a ridiculous 96 gb model running in tripple channel.

Yep, even if you had 8 x 16GB DIMMs installed and lot's of MemoryConfig entries, it's too low available space after a forced garbage collection.

So already some permanent damage done which is unreversable?

Very possible, with too low free space you force garbage collection to work constantly and this damages the SPI flash memory NAND cells. When the cells fail, you have a brick.

I'm gonna send you a PM, get everything exactly as instructed and I'll investigate what is really going on.
 
  • Like
Reactions: trifero

cMP 51 neewbie

macrumors newbie
Jan 9, 2022
18
4
Yep, even if you had 8 x 16GB DIMMs installed and lot's of MemoryConfig entries, it's too low available space after a forced garbage collection.



Very possible, with too low free space you force garbage collection to work constantly and this damages the SPI flash memory NAND cells. When the cells fail, you have a brick.

I'm gonna send you a PM, get everything exactly as instructed and I'll investigate what is really going on.
thank you for looking into this!
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
Who would bet that one more EFI release would be found and dumped after all these years? Better yet, one earlier than all others!

Yes, thanks to @Edge, now we have the dump and the data for MP41.0081.B03:

EFI Release vs BIOS Version vs BootBlock Version Reference Table:

EFI Release:BIOS Version:BootBlock Version:
MP41.0081.B03MP41.88Z.0081.B03.0902231259AAPLEFI1.88Z.0004.I00.0901121311 (this could be from an earlier BootROM)
MP41.0081.B04MP41.88Z.0081.B04.0903051113AAPLEFI1.88Z.0004.I00.0901121311 (this could be from an earlier BootROM)
MP41.0081.B??MP41.88Z.0081.B??AAPLEFI1.88Z.0004.I00.0908061259
MP41.0081.B07MP41.88Z.0081.B07.0910130729AAPLEFI1.88Z.0004.I00.0910130728
MP41.0081.B08MP41.88Z.0081.B08.1001221313AAPLEFI1.88Z.0004.I00.1001221311
MP51.007F.B00MP51.88Z.007F.B00.1008031144AAPLEFI1.88Z.0005.I00.1006041028
MP51.007F.B??MP51.88Z.007F.B??AAPLEFI1.88Z.0005.I00.1007141219
MP51.007F.B01MP51.88Z.007F.B01.1008231310AAPLEFI1.88Z.0005.I00.?
MP51.007F.B02MP51.88Z.007F.B02.1009221128AAPLEFI1.88Z.0005.I00.?
MP51.007F.B03MP51.88Z.007F.B03.1010071432AAPLEFI1.88Z.0005.I00.1010071430
MP51.0083.B00MP51.88Z.0083.B00.1707271620AAPLEFI1.88Z.0005.I00.1707271617
MP51.0084.B00MP51.88Z.0084.B00.1708080528AAPLEFI1.88Z.0005.I00.1708080525
MP51.0085.B00MP51.88Z.0085.B00.1802021746AAPLEFI1.88Z.0005.I00.1802021742
MP51.0087.B00MP51.88Z.0087.B00.1804181525AAPLEFI1.88Z.0005.I00.1804181521
MP51.0089.B00MP51.88Z.0089.B00.1806081708AAPLEFI1.88Z.0005.I00.1806081704
138.0.0.0.0MP51.88Z.F000.B00.1807300628AAPLEFI1.88Z.0005.I00.1807300627
139.0.0.0.0MP51.88Z.F000.B00.1808171030AAPLEFI1.88Z.0005.I00.1808171029
140.0.0.0.0MP51.88Z.F000.B00.1809191555AAPLEFI1.88Z.0005.I00.1809191554
141.0.0.0.0MP51.88Z.F000.B00.1812191621AAPLEFI1.88Z.0005.I00.1812191620
142.0.0.0.0MP51.88Z.F000.B00.1902142049AAPLEFI1.88Z.0005.I00.1902142048
144.0.0.0.0MP51.88Z.F000.B00.1904121248AAPLEFI1.88Z.0005.I00.1904121247

Also, this indicates that the BootBlock versions for MP41.0081.B03 and MP41.0081.B04 could be not correct and the Mac Pro was upgraded from a previous release - BootBlocks are never upgraded by efiflasher when the EFI is upgraded.

BootBlock version AAPLEFI1.88Z.0004.I00.0901121311 is probably from MP41.0081.B02 or something.
 
Last edited:

skodises

Contributor
Jan 9, 2021
41
46
Amazing work. I can pretty much guarantee that you now know more about the history of the guts of the bootblock than Apple does- the chances of any of the folks who worked on the really early stuff still being employed at Apple approaches zero...
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
Amazing work. I can pretty much guarantee that you now know more about the history of the guts of the bootblock than Apple does- the chances of any of the folks who worked on the really early stuff still being employed at Apple approaches zero...
Some weeks ago on Twitter, Chris Espinosa (Apple badge number 008 and the longest-serving Apple employee) wrote about the number of people that started to work for Apple in the late 70's that still work there today. If I remember correctly, around 2 or 3 dozen employees from the early days still work there, several of them uninterruptedly.

With the release of MacPro4.1 in March 2009, the design phase likely started around the second quarter of 2008, the official release date for Nehalem is November 11, 2008, so around 13 years ago. Probably many people who worked on hardware and firmware design still work there.

Btw, the last MacPro5,1 EFI firmware update was back in April 2019, so it's not something so remote.
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
How to do a BootROM dump with ROMTool:

  • If you are running anything newer than Mavericks, you will need to disable SIP. You can boot your Recovery partition or you can boot a createinstallmedia USB installer (≥10.11) to disable SIP.

    Boot your Recovery partition or boot a createinstallmedia USB installer, open Terminal and then disable SIP with the command:
    Code:
    csrutil disable
    Note: Yosemite SIP is not compatible with ROMTool/flashrom. Don’t use Yosemite at all.
  • Do a BootROM dump using ROMTool, zip password is "rom". You need SIP disabled and no AV or any anti-malware running. ROMTool is usually a false-positive to any AV/anti-malware because it uses flashrom and DirectHWAccess.kext. If you have any AV/Malware detection tool installed, do a clean install on another disk, it will be easier and less time consuming than disabling/removing kexts/etc.

    If ROMTool asks you to confirm what is the model of your SPI flash, it's the 8-pin SOIC flash memory next to the PCIe AUX-B power connector, label U8700 - see the photos below. The model of the SPI flash memory is usually related to the model year:​
    • with early-2009 almost all backplanes have SST25VF032B,
    • with mid-2010 usually is Macronix MX25L3205A or MX25L3205D, sometimes can be MX25L3206E with mid-2010s made in the 1st half of 2012, very rarely is SST25VF032B,
    • with mid-2012 usually is Macronix MX25L3206E, it's not common to see mid-2012 with MX25L3205A or MX25L3205D.
    • If ROMTool don’t ask you the SPI model at all, Apple used a SST25VF032B.
 

Attachments

  • MP51 - A - LBSN_MLB - SPI.jpeg
    MP51 - A - LBSN_MLB - SPI.jpeg
    457 KB · Views: 1,883
  • MP51 - E - U8700.MX25L3205D.JPG
    MP51 - E - U8700.MX25L3205D.JPG
    416.2 KB · Views: 369
  • MP51 - F - U8700.SST25VF032B.JPG
    MP51 - F - U8700.SST25VF032B.JPG
    423.6 KB · Views: 369

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601

How to check the health of the NVRAM/if the garbage collection is still working:​


There is a extremely simple way (simple here as in not need to know how the NVRAM works/check free space indicators/validate checksums and etc) to check if the garbage collection failed and the need to reflash with the clean dump:

  • Dump the BootROM with ROMTool
  • Open the dump with UEFITool NE A63, the most recent release of UEFITool NE that is compatible with MacPro5,1 BootROM images.
  • Go to EFISystemNvDataFvGUID, open it
    vss_44781-efisystemnvdatafvguid-png.1730048
    vss_44781-efisystemnvdatafvguid-png.1790554
  • Go to the first VSS store, open it:
    vss_44781-efisystemnvdatafvguid-vss-png.1730029
  • Click Free space, it's after the last variable/VSS entry:
    vss_44781-efisystemnvdatafvguid-vss-free-space-png.1730028
  • Check on the right panel the Full size:
    vss_44781-efisystemnvdatafvguid-vss-free-space-full-size-png.1730027
A clean reconstructed never booted image have the Free Space Full Size as 65448 - this is for a fully empty store:
vss_reconstructed-empty-png.1730012


A normal working single CPU Mac Pro with 3 DIMMs have the Free Space Full Size usually around 45000 to 40000 - this is for a healthy working dump:
vss_44781-efisystemnvdatafvguid-vss-free-space-full-size-png.1730027


A normal working dual CPU Mac Pro with 8 DIMMs (DIMM configuration data and SPD caches stored by MemoryConfig NVRAM entries are what takes the most space inside the VSS store) have the Free space Full size around 35000 to 30000 - this is for a healthy working dump:
vss_34808-png.1730024


A Mac Pro that the garbage collection is not working anymore will have less than 1/3 of Free Space Full Size available, less than 22000 bytes available. Any less than this and you usually start having problems and could even brick your Mac Pro. This is critical with Big Sur and Monterey OTA Software Updates, but also happened in the past with previous macOS versions (examples #1, #2) because the number of variables saved while staging the software updates.

This one has just 8921 and already corrupted the NVRAM volume:
vss_8921-png.1730020


Examples of corruption:​


Where is the secondary VSS store?
vss_8921_no-2nd-vss-png.1730021


This dump below had two different failures, a corrupt circular log and failed garbage collection on primary VSS store where after the corrupt point the circular log was identified as padding, and the secondary VSS store completely trashed and not even being identified by UEFITool anymore. The owner of this early-2009 got it repaired in the nick of time before bricking it.
screen-shot-2022-01-11-at-08-24-51-png.1942311


If you found that your NVRAM volume have any of the issues above and you need a BootROM reconstruction service, send me a PM.


Advice after several bricks over the years:​


First a fact, MacPro5,1 NVRAM was designed back in 2008ish, when the NVRAM was used sparingly. Now we are in 2022 and the NVRAM is used constantly for all sort of things, like all sort the iCloud variables (for example, the Wi-Fi credentials for the wireless networks that you connect with your iPhone and MacBooks are also saved inside the Mac Pro NVRAM) to the several variables needed to bootstrap software updates when you have sealed containers (BigSur and Monterey).

The NVRAM is now the Achilles heel of our MacPro5,1 and I personally don't wait for the garbage collection to fail. Now I have a recurring appointment on my Calendar to flash the never booted BootROM image every 3 months. Since starting doing it, I never had a brick or any NVRAM problems - even with all my crazy tests that bricked so much times my backplanes in the past. Do the same.

Btw, flashing a clean dump is a process that is a lot less wear intensive to the NAND cells than the whole garbage collection process. Only the sectors that need to be erased/re-written will be when you flash the clean dump, while the garbage collection process have to copy the valid circular log to the secondary VSS store, erase the primary, write to it, erase the secondary and etc.[/LIST]
 
Last edited:

retlif

macrumors member
Feb 2, 2020
79
34
I checked my computer and my "Free Space Full Size" is 24597. Its single CPU Mac Pro (6-Core) wit 3 DIMMs 16gb.

So, I understand that I should try to deep nvram reset?

I am using opencore from Martin Lo package on SATA SSD that runs Big Sur on NVMe. I don't have any other operations systems installed (in bootpicker are only Recovery and Time Machine). I would like to know what to expect after reboot?

I understand that I will lose the bootpicker? Or is there a chance that the computer will boot from OC disk (SATA) anyway? What about SIP settings? Will be re-enabled?

How to bring the system back to life when it does not recognize OC? All you need to do is bless the Opencore disk with:

sudo bless --verbose --file /Volumes/VOLNAME/EFI/OC/OpenCore.efi --folder /Volumes/VOLNAME/EFI/OC --setBoot

Or? Are there any additional surprises waiting for me?

I understand that Handoff, iCloud and all the rest of the stuff stored in NVRAM will gone and need to be configured again?


TIA
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
I checked my computer and my "Free Space Full Size" is 24597. Its single CPU Mac Pro (6-Core) wit 3 DIMMs 16gb.

So, I understand that I should try to deep nvram reset?
Correct, check the main VSS space available again after the deep NVRAM reset.
I am using opencore from Martin Lo package on SATA SSD that runs Big Sur on NVMe. I don't have any other operations systems installed (in bootpicker are only Recovery and Time Machine). I would like to know what to expect after reboot?

I understand that I will lose the bootpicker?
The deep NVRAM reset will clear the boot variables.
Or is there a chance that the computer will boot from OC disk (SATA) anyway?
Usually boots from SATA bay one, not guaranteed.
What about SIP settings? Will be re-enabled?
Yes.
How to bring the system back to life when it does not recognize OC? All you need to do is bless the Opencore disk with:

sudo bless --verbose --file /Volumes/VOLNAME/EFI/OC/OpenCore.efi --folder /Volumes/VOLNAME/EFI/OC --setBoot

Or? Are there any additional surprises waiting for me?

I understand that Handoff, iCloud and all the rest of the stuff stored in NVRAM will gone and need to be configured again?


TIA
You will need to re-bless your OC ESP, see the OpenCore thread first post.
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
@retlif

Forgot to add, bless your OC ESP only AFTER you dump your BootROM image again. Never dump/flash your BootROM image booted from OC, boot your Mojave rescue disk to do that.
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
Ok but can I ask why?
Several cases in the past that the NVRAM part of the BootROM image was not correctly dumped when dumped from unsupported installs, don't risk it, boot native when interacting with the BootROM.
 

Warrington

macrumors member
Dec 13, 2021
69
21
A Mac Pro that the garbage collection is not working anymore will have less than 1/3 of Free Space Full Size available, less than 22000 bytes available. Any less than this and you usually start having problems.

This is not exactly true. After an NVRAM deep reset, the 2nd VSS store can stay empty for a while and the 1st VSS store free space will decrease to 20000 bytes or even to 12000 bytes.
 

tsialex

Contributor
Original poster
Jun 13, 2016
13,454
13,601
This is not exactly true. After an NVRAM deep reset, the 2nd VSS store can stay empty for a while and the 1st VSS store free space will decrease to 20000 bytes or even to 12000 bytes.
When the secondary VSS store is empty/don't have a copy of the valid entries of the circular log, can be that the garbage collection wasn't needed at all (not the case when the available space is too low) or that it was interrupted before the valid circular log entries were copied - the former is a signal of problems.

When your available space is below 1/3 after a deep NVRAM reset, it's never a good sign and you have to track it. To have a VSS store without space is a real problem when macOS set up the variables needed to bootstrap software updates with BigSur and Monterey.

Also, for people that flashed a never booted image, dumping it right after is not meaningful. The circular log is being repopulated and can give erroneous results, I've written about this exact issue on the OC thread, my recommendation is to follow the available space after a week.
 
Last edited:
  • Like
Reactions: JedNZ and trifero
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.