2019 16" MacBook Pro (A2141) macOS install issues. NAND? Wifi? Firmware?

[TechnoMancer]

Hi All.


Sorry ahead of time for the text wall. This is a strange problem and I have tried to explain it as best I can. At this stage I am not sure if this is a hardware or firmware issue and hoping that someone more knowledgeable than me in the area of understanding installation logs, or someone that has had a similar issue in the past may have some ideas of what might be causing this or how to fix it.


The issue:
Unable to do a clean install of any macOS versions supported for this model, even after fully erasing and formatting the internal SSD, DFU revives/restores, and fails regardless of method used for installation. Install process dies with different errors at different stages (depending on macOS version being installed and installation method).

Background of the machine:
An A2141 16” 2019 MacBook Pro (820-01700 logic board with 1TB onboard SSD) that is having issues with installing macOS (any of the supported versions for this model) onto its internal (onboard) SSD. The client tried to do an upgrade to Monterey (not sure what it was running before that), and the upgrade process apparently died part way through, and the machine has been unable to successfully install any macOS since. The board is pristine, runs well, and without any noticeable issues. No signs of liquid damage or corrosion, and not even a speck of dust to be seen. It came to me as a hardware fault with suspected failed onboard SSD (NAND chips) I know that this model is renowned for failing NAND chips, however after inspecting the board and doing some measurements and software related tests I am not convinced that the issue is hardware related. I still think that it looks more like firmware, disk encryption, or possibly even network related? I could be wrong of course, and it could very well still be NAND chips, but it doesn’t seem to behave in the same way as others I have worked on with failed NAND chips.


What I have tried so far:

• At first it sounded like corrupt firmware to me, so ran several revives and restores in DFU mode. All complete successfully, however the issue persists when attempting to reinstall macOS.
• Have tried installing Catalina, Big Sur, Monterey, and Ventura and all fail with similar errors (will attach installation logs for some to this post).
• Have tried installing from external USB installer to internal SSD using all macOS versions mentioned above. All error out at different stages but all do error out.
• Have tried installing from external USB installer to external HDD/SSD (which should bypass internal SSD completely?) using all macOS versions mentioned above which also all error out with the same messages.
• Have tried installing using internet recovery (+Shift +Option +Command +R) which installs Catalina, and installation fails with similar errors.
• Have tried installing using internet recovery (+Option +Command +R) which installs Ventura, and installation fails with similar errors.
• Have tried cloning above macOS versions from previously installed ext. disks to int. SSD using Carbon Copy Cloner. Only Catalina boots and runs properly afterwards from the internal SSD. (Monterey onwards isn't supported anymore by CCC as far as I know when cloning a boot drive?)
• Can boot and run perfectly fine and stable from any external disks that are preinstalled with any of the macOS versions mentioned above, including Monterey (haven't tested Ventura yet on external disk). Sadly I am not able to do that at the moment any longer, as since my last DFU Restore, the default Startup securoty option ("Don't allow booting from external drives") is now re-enabled, and I can't disable until I can get an OS on the SSD again (no admin account at present).
• Internal SSD seems to work perfectly well and is not reporting any errors when formatting or being accessed from other instances of macOS. SMART status of internal SSD is listed as "verified" so looks to be fine SMART wise.
• I have made sure that SIP is enabled, and also set the start-up security "Secure Boot" option to "Full Security" - have also tried "No Security" option though to no avail as well.
• I have zeroed the first part of the internal SSD using dd and writing /dev/zero to it to destroy all partitions. This didn't help.
• Have run diagnostics (Holding “D” key), and no issues found.

The errors being seen:
I will upload the Ventura installation log, as well as the Catalina installation log. All were done from Internet Recovery. Ventura and Monterey both show an exceptionally useful message when they fail: "An error occurred loading the update.". Catalina installation dies with a message saying “An error occurred installing macOS” then wants me to run diagnostics.
There also seems to be a lot of DHCP errors in the install logs. Not sure if that is normal. I did manage to get 2 Catalina installations to actually complete successfully. I changed the DNS servers in my DHCP scope to 1.1.1.1 during testing at one point, and then when I tried an Internet Recovery I had successful installations (twice only), however this only worked twice, so could have been total coincidence. Fact remains though that it does seem to intermittently install successfully.

The lines that appear most important to me in the Ventura installation log:

Quote:

Oct 26 12:33:45 MacBook-Pro osinstallersetupd[478]: **** /Volumes/Macintosh SSD/macOS Install Data/UpdateBundle/AssetData/./boot/Firmware/018-26018-570.dmg.x86.mtree != ./boot/Firmware/018-26018-570.dmg.x86.mtree **** Oct 26 12:33:45 MacBook-Pro osinstallersetupd[478]: Operation queue failed with error: Error Domain=com.apple.OSInstallerSetup.error Code=1004 "An error occurred loading the update." UserInfo={NSLocalizedDescription=An error occurred loading the update.}

Does that not imply that it was unable to update some firmware? Or is it comparing the installed firmware with the version in the installer and saying it is not the same ("!=")? Either way, the installation seems to halt right after that.

And the most important lines in my opinion from the Catalina logs which also seem to be firmware (bless) related:

Quote:

Oct 26 05:26:08 MacBook-Pro OSInstaller[637]: Failed to get disk for UUID specified for rebless. Oct 26 05:26:08 MacBook-Pro OSInstaller[637]: ------- Install Failed ------- Oct 26 05:26:08 MacBook-Pro OSInstaller[637]: Operation: Install packages failed, Failure Reason: Error Domain=PKInstallErrorDomain Code=110 "An error occurred installing macOS. To use Apple Diagnostics to check your Mac hardware, shut down, press the power button, and immediately hold the ‘D’ key until diagnostics

I am stumped and have tried just about everything I can think of. If anyone can see anything in the logs that might point me in the right direction to get this fixed, I would be eternally grateful! The area I usually play is in fixing logic boards at component level, and I am way out of my depth when trying to decipher installation logs lol.

The only other thing I plan to try to attempt to eliminate the network side of things as a cause, is I have ordered a Thunderbolt 3 to Gigabit Ethernet adapter which should arrive on Friday. With this I plan to try to do an Internet Recovery over the wire rather than over the air in case the machine has some sort of wifi related logic board fault causing all of this weirdness.

 

[Fishrrman]

Apple Configurator II, run from another, working Mac.
You need to connect the working and non-working Macs with a USBc cable.

 

[chrfr]

Apple Configurator II, run from another, working Mac.
You need to connect the working and non-working Macs with a USBc cable.

This won't reinstall the operating system on an Intel Mac, and the OP has already done the only option in Configurator for these Macs, which is to restore the T2 firmware.

 

[chrfr]

The lines that appear most important to me in the Ventura installation log:

Quote:

Oct 26 12:33:45 MacBook-Pro osinstallersetupd[478]: **** /Volumes/Macintosh SSD/macOS Install Data/UpdateBundle/AssetData/./boot/Firmware/018-26018-570.dmg.x86.mtree != ./boot/Firmware/018-26018-570.dmg.x86.mtree **** Oct 26 12:33:45 MacBook-Pro osinstallersetupd[478]: Operation queue failed with error: Error Domain=com.apple.OSInstallerSetup.error Code=1004 "An error occurred loading the update." UserInfo={NSLocalizedDescription=An error occurred loading the update.}

Does that not imply that it was unable to update some firmware? Or is it comparing the installed firmware with the version in the installer and saying it is not the same ("!=")? Either way, the installation seems to halt right after that.

I don't see anything there that would necesarily point to firmware failing; the T2 restore would update the firmware to the newest version available anyway. Does your network have unrestricted access to Apple?
I think the most likely suspect is your wireless network. Otherwise, I might bring the computer to an Apple store if you have one nearby to have them run diagnostics and try to reinstall the OS on their network.

 

[tinygoblin]

Have you wiping internal disk device (via Terminal) and installing Windows 10 via installation media (USB flash drive)? You can never be sure what's going on if you use inferior software working through Internet, use the real deal product. If everything goes well all you have to do is say goodbye to gayOS with it's quirks and features.

P.S. Get Boot Camp support software via

brigadier --model MacBookPro16,1

@ GitHub and just place $WinPEDriver$ folder to the root of your Windows 10 installation media, it'll get the things going (other drivers and Boot Camp Control Panel can be installed later on once installation is complete).

 

[TechnoMancer]

I don't see anything there that would necesarily point to firmware failing; the T2 restore would update the firmware to the newest version available anyway. Does your network have unrestricted access to Apple?
I think the most likely suspect is your wireless network. Otherwise, I might bring the computer to an Apple store if you have one nearby to have them run diagnostics and try to reinstall the OS on their network.

Thanks. The external TB3 to gigbit ethernet RJ45 adapter arrives tomorrow, so I will try again using that instead of wifi and see if it behaves any differently. My network is just regular home network. Fibre on the backend with a standard issue firewall router and its out of the box "firewalling", so no restricted access as such from the network layer. Only thing that could be counted as "restricted" would be that I usually use OpenDNS servers in my DHCP scopes on the router. Their servers are much slower than Cloudflare (1.1.1.1) or Google (8.8.8.8). When I changed the scopes to use the Cloudflare and Google servers, I actually had a couple of successful installations which was weird. I know that the machines phone home a few times during an installation checking various things, and doing encrypty stuff, so perhaps if those calls fail the installation dies. Heres hoping lol.

The machine was booked in with me as it was suspected to have a hardware fault (I repair logic boards). If all of this fails to show anything as the cause I might have to tell the client to take it to Apple and see what they say, as other than trying to do internet recovery using a cable to prove if the issue is related to the wifi/bluetooth chip, I am not sure that there's any other way for me to prove what is at fault.

 

[TechnoMancer]

So after much testing and troubleshooting, it looks like the issue is either onboard RAM chip failure, or CPU L2 cache failing causing the OS installation to fail.

Does anyone know how to interpret Memtest results (see attached images)? To me, it looks like RAM is faulty based on the address ranges where the fault is reported, but it seems to be running tests on the L2 Cache at the time. Don't want to tell the customer it's an onboard RAM chip fault (which I can repair) if it is actually failed CPU L2 cache.

Anyone able to decypher the results or know what these results mean?

 

[GumaRodak]

Perhaps Factory reset and then update?

 

[TechnoMancer]

Perhaps Factory reset and then update?

Thanks. Done that several times and no dice. At this stage it looks like either a faulty onboard RAM chip (which I can fix by chip replacement), or faulty CPU cache based on test results above, but my mind is telling me onoboard RAM chip, as CPU core seems to change between errors, so makes me think RAM is more likely. Hoping someone with Memtest experience can point me in the right direction.

 

[chrfr]

So after much testing and troubleshooting, it looks like the issue is either onboard RAM chip failure, or CPU L2 cache failing causing the OS installation to fail.

Does anyone know how to interpret Memtest results (see attached images)? To me, it looks like RAM is faulty based on the address ranges where the fault is reported, but it seems to be running tests on the L2 Cache at the time. Don't want to tell the customer it's an onboard RAM chip fault (which I can repair) if it is actually failed CPU L2 cache.

Anyone able to decypher the results or know what these results mean?


View attachment 2104895



View attachment 2104893 View attachment 2104894

There’s really nothing to decipher except that the computer has faulty RAM.

 

[TechnoMancer]

There’s really nothing to decipher except that the computer has faulty RAM.

Thanks, but from the test results it could also be interpreted as faulty L2 cache. That's what I mean by deciphering. Is it faulty RAM, or faulty CPU L2 cache?

 

[Bripro]

Did you fix it? I have a same model MacBook with the exact same problem

 

[TechnoMancer]

Did you fix it? I have a same model MacBook with the exact same problem

No, but established that it must one of the memory chips that is faulty. Did you get the same results when testing with Memtest86?

I quoted the client for fixing it, but they saadly decided not to go ahead with the repair. It would involve replacing the memory chips one at a time from a donor board until the error is no longer seen.