Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
Contrary to popular belief, this is a perfectly healthy NVRAM of a 4,1->5,1 machine:

nbxd144_22_1stVSS.png


test-nvram.app report on the same dump:

nbxd144_22.png


More details will follow.
 
  • Like
Reactions: bernuli

amstel78

macrumors 6502a
Aug 12, 2018
517
183
Interested to see the rest of this assertion as this contradicts findings from @tsialex. The amount of free space left in your screenshot above is dangerously low.

This is what I'm seeing after tonight's ROM dump. I'll need to reflash with a pristine ROM as it's been a while. That said, my 5,1 has all of its DIMM slots populated and is running OpenCore.
1643850249399.png


And after the 5th NVRAM reset chime. Looks like the garbage collector is still working fine.
1643852709328.png
 
Last edited:

Dayo

macrumors 68020
Dec 21, 2018
2,256
1,278
Last edited:

Fastsavage

macrumors regular
Jun 28, 2011
178
40
New Zealand
It’s pretty well documented that the NVRAM is the weakest point of our 4,1 and 5,1 so we need to take care of them and use advise and experiences from credible sources.

We all know that with computers you see new errors and faults every day… always something the same but different.

My vote is with experience.
 
  • Like
Reactions: prefuse07

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
The amount of free space left in your screenshot above is dangerously low.

The amount of free space is actually changing at every reboot, this is the little "secret" nobody will tell.

Here is the free space after the next reboot. Now it looks great, right?

nbxd144_23_1stVSS.png


And the same amount of free space, as reported by test_nvram.app:

nbxd144_23.png


This is how NVRAM really works, 1st VSS store free space is never static. Only the 2nd VSS store is sometimes static.
 

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
My vote is with experience.

My experience is now based on analyzing hundreds of dumps from dozens of 5,1 and 4,1->5,1 machines.

8 reboots (and 8 dumps) before, the 1st VSS store free space was reported the same:

nbxd144_15.png


And this is how things changed after the next reboot:

nbxd144_16.png


With absolutely nothing changed on the machine, in terms of hardware or software, 3 memory configs have been added and 7040 Bytes of free space have been used.
 
Last edited:

KeesMacPro

macrumors 65816
Nov 7, 2019
1,453
595
My experience is now based on analyzing hundreds of dumps from dozens of 5,1 and 4,1->5,1 machines.
I'm curious about the way you're analyzing these dumps.
I suppose you're not only running the Macschrauber's dumper and looking at the amount of free space...
 

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
I'm curious about the way you're analyzing these dumps

UEFITool NE is a great tool and most of the results won't go public. Binwalk is also a great tool, but is not the beginning and the end of all things.

The screenshots from Macschrauber's dumper are easier to understand for everybody, this is why I include them in these posts.

I have some very interesting results from a 5,1 mid-2010 machine and I will write about them within a few days.
 

amstel78

macrumors 6502a
Aug 12, 2018
517
183
The amount of free space is actually changing at every reboot, this is the little "secret" nobody will tell.
The amount of free space changing on the SPI flash after each reboot is normal behavior, as far as I understand. This can be affected by the garbage collection routine. The problem arises when certain operations require many variables to be stored in the flash ROM (such as during an OS update), and there's not enough available space remaining. With your first screenshot showing less than 4k bytes, what happens if that does not free up after a reboot during an OS update? I assume it could lead to corruption elsewhere in the ROM.
 

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
With your first screenshot showing less than 4k bytes, what happens if that does not free up after a reboot during an OS update?

I don't know what happens if does not free up. What I know for sure is it does free up every time, this is what I'm writing about.

After every reboot, the free space decreases with something between 7000 and 8000 Bytes. And when the remaining free space of the 1st VSS store is less than that (at the 8th reboot for this particular machine) it does free up. This is how the NVRAM really works, this is the point.

The other point is the machine won't free up more space if it "believes" that more space is not needed (between the 4th and the 5th reboot, let's say).
 

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
The problem arises when certain operations require many variables to be stored in the flash ROM (such as during an OS update), and there's not enough available space remaining.

This is what I used to believe a few months ago. That 5,1 mid-2010 machine mentioned above helped me learn a few things.
 

amstel78

macrumors 6502a
Aug 12, 2018
517
183
After every reboot, the free space decreases with something between 7000 and 8000 Bytes. And when the remaining free space of the 1st VSS store is less than that (at the 8th reboot for this particular machine) it does free up. This is how the NVRAM really works, this is the point.
Interesting. Well, I myself don't know too much about the inner workings of the SPI flash ROM, but I am concerned about it from a longevity perspective. I'm interested to read more about the behavior you posited above, as this is the first time I've heard about there being a specific, repeatable pattern to ROM memory management.
 

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
I am concerned about it from a longevity perspective.

Same here. In my case, I'm concerned about the longevity perspective of dozens of 4,1->5,1 machines, so I ended up doing this research. One of the interesting results is that the SST25VF032B from the 4,1 machines actually seems to generate less problems than the MX25L3205D from the mid-2010 machines.
 

Dayo

macrumors 68020
Dec 21, 2018
2,256
1,278
This is what I used to believe a few months ago.
So what happens if you are running an update that needs to store large data and you are on the 6th or 7th reboot of the garbage collection cycle for instance? Are you saying the firmware will realise it needs more space in advance and trigger the garbage collection even though it would only have ordinarily done this on the 8th reboot?

I suppose what you are pointing out is not generally known and I get your point that it hasn't perhaps been explained as it perhaps should have.

An important point that should be stressed, but perhaps hasn't enough, is that you should always do a triple NVRAM reset to force garbage collection BEFORE running any update.

this is the first time I've heard about there being a specific, repeatable pattern to ROM memory management.
There hasn't been a lot of focus on this but this is how the NVRAM works.

There is one example where it is set out:

EDIT: Changed "is generally known" to "is not generally known"
 
Last edited:

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
So what happens if you are running an update that needs to store large data and you are on the 6th or 7th reboot of the garbage collection cycle for instance? Are you saying the firmware will realise it needs more space in advance and trigger the garbage collection even though it would only have ordinarily done this on the 8th reboot?

I'm not saying the firmware will realise anything, I will tell everybody exactly what happened. Was nothing dramatic.
 
  • Like
Reactions: NC12

amstel78

macrumors 6502a
Aug 12, 2018
517
183
There hasn't been a lot of focus on this but this is how the NVRAM works.

There is one example where it is set out:
Thanks. I just figured the garbage collector would run after each reboot, or if a set threshold was reached. I didn't realize it would take up to 8 reboot cycles for the memory to be dumped.
 
  • Like
Reactions: NC12

Dayo

macrumors 68020
Dec 21, 2018
2,256
1,278
I didn't realize it would take up to 8 reboot cycles for the memory to be dumped.
Seems to vary. Was the 10th reboot in the linked example.

Main point is that you would not ordinarily know where things are in terms of free space or what payload size needs to go into the NVRAM. If things are not in your favour, you can expect to get an overrun and ultimately, a brick.

This is of course unless we actually ever get to see the mentioned "nothing dramatic" outcome at some point.
 
Last edited:
  • Like
Reactions: amstel78

Macschrauber

macrumors 68030
Dec 27, 2015
2,976
1,483
Germany
@Warrington I wonder if you use the latest version of my Tool.




As I don't see the notification about VSS2 being empty after triggered garbage collection.

And yes, we know the NVram gets filled up until almost full. After full the garbage collection will run. This will take about 12 reboots from almost empty to full, see the screenshots Dayo posted the link for.

A Dual machine with lots of Ram and 8 Memory slots filled up have bigger nvram "packets" as a Single CPU machine with just 2 Ram slots populated. There are some more variables filling the nvram like stored wifi networks and much more. So it depends how much reboots it takes to get the garbage collection run.


@tsialex investigated and posted loads of details. His investigation and his help made me able to start writing my dumper as I wanted a tool for our CMPs.

Plus I wanted people to take more care about the boot rom and to make backups.
 
Last edited:

Macschrauber

macrumors 68030
Dec 27, 2015
2,976
1,483
Germany
... and btw

I have also about 300 dumps and use them for mass tests if I add a feature to the bash script behind the dumper.

If you want to use it for tests you can either extract the test_nvram bash script or use it like (example is for the mounted .dmg)

Code:
. /Volumes/Macschrauber\'s\ CMP\ Rom\ Dump/RomDump\ Macschrauber.app/Contents/Resources/test_nvram ~/ck.bin
serial from firmware: CK2xxxx
MP51.88Z.0089.B00.1806081708
25 Memory Configs (ok)
2 xml (ok)
0 iCloud Tokkens (ok)
0 Microsoft Certificates (ok)
2 BluetoothActiveControllerInfos (ok)
2 BluetoothInternalControllerInfos (ok)
2 current-network (ok)
9 AAPL Path Properties (ok)
Length of 2nd VSS Store is wrong (FF FF FF FF)
7616 Bytes free space of 65472


for cutting the serial number the argument -shortserial can be added to test_nvram
 

Warrington

macrumors member
Original poster
Dec 13, 2021
69
21
I don't see the notification about VSS2 being empty after triggered garbage collection.

VSS2 is not empty, it stays at 57813 Bytes of free space during the entire cycle of 8 reboots:

nbxd144_23_2ndVSS.png


VSS2 was empty only for the first 4 reboots after the deep NVRAM reset and your tool was correctly reporting it as empty. After the 4th reboot, the VSS2 was still empty, but the free space of the 1st VSS was already below 7000 Bytes:

nbxd144_06.png


That recommendation (cleanup/triple nvram reset) was not necessary at that point, the triple nvram reset itself created that situation and no other deep reset was needed.

The next reboot cleaned the 1st VSS completely:

nbxd144_07.png
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.