MP 1,1-5,1 Mac OS 11.3 has broken support for older Mac Pros

[JohnD]

Yesterday, by accident, my boot issues vanished. I’ve now booted into 11.3.1 11x times in a row and counting.

I once booted 11x fine, then the 12th hung. The other day I booted 22x fine, then the 23rd failed, as did many subsequent reboots. 11x simply isn't enough to say your boot issues vanished - they've likely just been delayed. Now, if anyone can claim 30x-50x reboots in a row and no issues, that'd be interesting...

 

[MacForumsUser]

I can’t be of much help, so I’ll cheerlead a bit. I just want to say a BIG THANK YOU to everyone working so incredibly hard to track down the source of this nasty issue and fix it. You are superstars, rockstars, Homestars…well…STARS! I - and I’m sure many others waiting patiently on bated breath - are immensely thankful and appreciative of the work you’ve done, are doing and will do to figure out this problem…and we know you will. Godspeed!

 

[startergo]

at the moment are /S/L/E/AppleSMC.kext

Replacing this leads to a constant panic at boot.

 

[cdf]

If the issue is related to the SMC, I wonder if enabling AppleSmcIo in OC could help.

 

[startergo]

There is some difference in macOS 11.4 (20F5055c).
It hangs consistently with NVME assert (no panic though) with a presence of an NVME drive.
Boots normally with 6 drives, one of which is 11.2.3 BS (2 PCIE SATA drives):

boot-args="-v -no_compat_check keepsyms=1 debug=0x144 -liludbgall -nvmefaspm"

 

[JohnD]

There is some difference in macOS 11.4 (20F5055c).
It hangs consistently with NVME assert (no panic though) with a presence of an NVME drive.
Boots normally with 6 drives, one of which is 11.2.3 BS (2 PCIE SATA drives):

Can you elaborate on "boots normally"? 11.2.3 or earlier I assume (and not 11.3)? I installed 11.4b1 (and now b2) on an HDD and found the percentage of successful boots to be much lower than 11.3.x. Throughout the 11.3 beta cycle, I had a ~99% success rate booting from the HDD, but once upgraded to 11.4, that dropped to ~75%.

 

[JohnD]

Is it worth exploring downgrading the firmware from 144 to High Sierra's MP51.0089.B00? Maybe something in Apple's NVMe code is causing the issue? Just trying to think about all the machines that DO work fine, like all the hackintosh machines, and apparently the 2008 3,1 Mac Pro (of all things).

 

[startergo]

My two main areas of focus at the moment are /S/L/E/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext. (and possibly including the parent /S/L/E/IOAHCIFamily.kext/ itself).

I got this error:

sudo kmutil install --volume-root /System/Volumes/Update/mnt1/ --update-all                 
checking collections...
updated extension 'com.apple.driver.usb.AppleUSBVHCICommonRSM' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/AppleUSBCommon.kext/Contents/PlugIns/AppleUSBVHCICommonRSM.kext' (mach-o UUID changed from <unknown> to 072E85EC-FAED-36E2-B68B-159EC70E5119)
rebuilding release collections: boot, system
rebuilding release collections:
    boot kernel collection
    system kext collection
failed to build release collection:
    Unable to resolve dependencies: 'com.apple.driver.AppleAHCIPort' names a dependency on 'com.apple.iokit.IOAHCIFamily', which was not found.
Error Domain=KMErrorDomain Code=71 "Unable to resolve dependencies: 'com.apple.driver.AppleAHCIPort' names a dependency on 'com.apple.iokit.IOAHCIFamily', which was not found." UserInfo={NSLocalizedDescription=Unable to resolve dependencies: 'com.apple.driver.AppleAHCIPort' names a dependency on 'com.apple.iokit.IOAHCIFamily', which was not found.}

 

[Dayo]

'com.apple.driver.AppleAHCIPort' names a dependency on 'com.apple.iokit.IOAHCIFamily', which was not found.

Can't find com.apple.iokit.IOAHCIFamily in your updated setup or, more likely, the version specified in the info.plist of com.apple.driver.AppleAHCIPort

 

[startergo]

Can't find com.apple.iokit.IOAHCIFamily in your updated setup or, more likely, the version specified in the info.plist of com.apple.driver.AppleAHCIPort

I never touched those drivers. By the way S/L/E/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext looks identical.

 

[startergo]

According to this issue:

IGP causes NVMe Kernel Panic CSTS=0xffffffff · Issue #1193 · acidanthera/bugtracker

Let me start with the fact that this is not a bug in NVMeFix or Whatevergreen but this seems like the best place to document the issue. I have an Intel 9600K/H370 system that experiences kernel pan...

github.com

IGP causes NVME Kernel Panic.
Something completely unrelated manifests as an NVMe error.

The IGP going into a low-power mode seems to disrupt power to the NVMe, causing it to crash/reset, and thus causing the timeout.

Something similar happens in our case although not related to IGPU.

 

[khronokernel]

I never touched those drivers. By the way S/L/E/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext looks identical.

Are you editing by terminal or by Finder? Reason I mention this is editing via Finder has been known to botch SLE on our end during testing with acceleration patches, otherwise files such as AppleCamera.kext and such would fail to link with errors like "Cannot find IOSurface", etc when clearly present on disk

 

[startergo]

Are you editing by terminal or by Finder?

through finder

 

[startergo]

No error through terminal:

sudo rm -R /System/Volumes/Update/mnt1/System/Library/Extensions/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext
Password:
bigsurssd@BigSurSSDs-Pro ~ % sudo cp -R /Users/bigsurssd/Desktop/IOAHCIBlockStorage.kext  /System/Volumes/Update/mnt1/System/Library/Extensions/IOAHCIFamily.kext/Contents/PlugIns/
bigsurssd@BigSurSSDs-Pro ~ % sudo chmod -Rf 755 /System/Volumes/Update/mnt1/System/Library/Extensions/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext
bigsurssd@BigSurSSDs-Pro ~ % sudo chown -Rf root:wheel /System/Volumes/Update/mnt1/System/Library/Extensions/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext
bigsurssd@BigSurSSDs-Pro ~ % sudo kmutil install --volume-root /System/Volumes/Update/mnt1/ --update-all
checking collections...
updated extension 'com.apple.iokit.IOAHCIBlockStorage' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext' (mach-o UUID changed from B44DB3DD-107C-3D76-BD17-066A232D8A2C to F7D35A58-F026-3A65-8DA5-9CD2152A52E4)
updated extension 'com.apple.driver.usb.AppleUSBVHCICommonRSM' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/AppleUSBCommon.kext/Contents/PlugIns/AppleUSBVHCICommonRSM.kext' (mach-o UUID changed from <unknown> to 072E85EC-FAED-36E2-B68B-159EC70E5119)
rebuilding release collections: boot, system
rebuilding release collections:
    boot kernel collection
    system kext collection
rebuilding local auxiliary collection
kmutil done
bigsurssd@BigSurSSDs-Pro ~ % sudo bless --folder /System/Volumes/Update/mnt1/System/Library/CoreServices --bootefi --create-snapshot

Not sure about this?

updated extension 'com.apple.driver.usb.AppleUSBVHCICommonRSM' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/AppleUSBCommon.kext/Contents/PlugIns/AppleUSBVHCICommonRSM.kext' (mach-o UUID changed from <unknown> to 072E85EC-FAED-36E2-B68B-159EC70E5119)

 

[startergo]

Still

Spoiler: panics

panic(cpu 12 caller 0xffffff80133be976): Kernel trap at 0xffffff8015aad131, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000000, CR3: 0x0000000017747000, CR4: 0x00000000000226e0
RAX: 0x00000000000000ff, RBX: 0x0000000000000000, RCX: 0x0000000000000000, RDX: 0x0000000000000009
RSP: 0xffffffa23467bd80, RBP: 0xffffffa23467bdb0, RSI: 0x00000009300d5ef3, RDI: 0xffffff94262ddf00
R8:  0xffffff9426302780, R9:  0xffffff875a2ca780, R10: 0x0000000000000001, R11: 0xffffff94262c3f00
R12: 0xffffff94262c3f00, R13: 0xffffff8015aac016, R14: 0x00000009300d5ef3, R15: 0xffffff94262ddf00
RFL: 0x0000000000010246, RIP: 0xffffff8015aad131, CS:  0x0000000000000008, SS:  0x0000000000000000
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0xc, PL: 0, VF: 1

Backtrace (CPU 12), Frame : Return Address
0xffffffa23467b7a0 : 0xffffff801328715d mach_kernel : _handle_debugger_trap + 0x3fd
0xffffffa23467b7f0 : 0xffffff80133cdfb3 mach_kernel : _kdp_i386_trap + 0x143
0xffffffa23467b830 : 0xffffff80133be5aa mach_kernel : _kernel_trap + 0x55a
0xffffffa23467b880 : 0xffffff801322ba2f mach_kernel : _return_from_trap + 0xff
0xffffffa23467b8a0 : 0xffffff801328697d mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffa23467b9c0 : 0xffffff8013286c73 mach_kernel : _panic_trap_to_debugger + 0x273
0xffffffa23467ba30 : 0xffffff8013a965ea mach_kernel : _panic + 0x54
0xffffffa23467baa0 : 0xffffff80133be976 mach_kernel : _sync_iss_to_iks + 0x2c6
0xffffffa23467bc20 : 0xffffff80133be65d mach_kernel : _kernel_trap + 0x60d
0xffffffa23467bc70 : 0xffffff801322ba2f mach_kernel : _return_from_trap + 0xff
0xffffffa23467bc90 : 0xffffff8015aad131 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController18GetOveragedRequestEy + 0x27
0xffffffa23467bdb0 : 0xffffff8015aac077 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController15CheckForTimeoutEP18IOTimerEventSource + 0x61
0xffffffa23467bde0 : 0xffffff80139ed585 mach_kernel : __ZN18IOTimerEventSource15timeoutSignaledEPvS0_ + 0xa5
0xffffffa23467be50 : 0xffffff80139ed486 mach_kernel : __ZN18IOTimerEventSource17timeoutAndReleaseEPvS0_ + 0xc6
0xffffffa23467be80 : 0xffffff80132cd745 mach_kernel : _thread_call_delayed_timer + 0x4a5
0xffffffa23467bef0 : 0xffffff80132ce6b4 mach_kernel : _thread_call_delayed_timer + 0x1414
0xffffffa23467bfa0 : 0xffffff801322b13e mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         com.apple.iokit.IONVMeFamily(2.1)[43E13757-B839-33F0-9002-5D2F6C945368]@0xffffff8015a9f000->0xffffff8015ac8fff
            dependency: com.apple.driver.AppleEFINVRAM(2.1)[F43510DC-B31C-3896-AC7C-163AE4115EC5]@0xffffff8014702000->0xffffff801470bfff
            dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[96361043-6668-3C9F-8B5B-01ED84DB7953]@0xffffff80148c4000->0xffffff80148d9fff
            dependency: com.apple.iokit.IOPCIFamily(2.9)[FF2862C4-7269-36A2-91B9-4E87D3B88713]@0xffffff8015d5d000->0xffffff8015d85fff
            dependency: com.apple.iokit.IOReportFamily(47)[93BE7431-4B8C-3D04-83F0-93254ADF8A5E]@0xffffff8015d94000->0xffffff8015d96fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[A78CAAA2-43A6-38EF-AEDA-3B957D358855]@0xffffff8015e60000->0xffffff8015e71fff

Process name corresponding to current thread: kernel_task
Boot args: -v -no_compat_check keepsyms=1 debug=0x144 -liludbgall -nvmefaspm

Mac OS version:
Not yet set

Kernel version:
Darwin Kernel Version 20.5.0: Fri Apr 23 23:00:08 PDT 2021; root:xnu-7195.120.53.121.1~1/RELEASE_X86_64
Kernel UUID: C410A9F5-44D8-3511-89D6-DBC0283D0795
KernelCache slide: 0x0000000013000000
KernelCache base:  0xffffff8013200000
Kernel slide:      0x0000000013010000
Kernel text base:  0xffffff8013210000
__HIB  text base: 0xffffff8013100000
System model name: MacPro5,1 (Mac-27AD2F918AE68F61)
System shutdown begun: NO
Panic diags file unavailable, panic occurred prior to initialization
Hibernation exit count: 0

System uptime in nanoseconds: 40931428989
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000000098a7ce2ee
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000028193d842c 0x0000000000000000
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleSMC

 

[startergo]

Overall BS11.4b2 has improved, and I can boot most of the times even with NVMe drives.

 

[startergo]

Overall BS11.4b2 has improved, and I can boot most of the times even with NVMe drives.

Funny thing is if there is a drive with 11.2.3, while booting 11.4b2 the kernel panic is supressed (never gets there) even though it halts on NVMe sometimes. Without it, it just panics sometimes and this is at the end even with updated IOAHCIBlockStorage.kext (from 11.2.3):

IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleSMC

 

[HuRR]

11.4b3 is out it seems.

 

[startergo]

Will test it soon:

 

[hutzi20]

- Mac Pro 4,1 with rebuilded ROM
- Single CPU X5675, 48GB RAM
- Pulse RX 580 8GB
- BCM94360CD Wifi/Bluetooth card
- no additional PCIe cards installed, no NVMe
- Apple Keyboard and Cinema Display connected via bluetooth
- Martins Package 0.6.9

I tried to boot a fresh installed version of 11.3.1 from an USB connected SSD. I tried it 8 times and had only 1 successful boot. The other times it Stucks all time at the same place.

Below you see the screen from two different tries. It's not much, but I think little things will help too.

 

[JohnD]

11.4b3 is out it seems.

@startergo beat me to it, but already installing it... on a hard drive... so I will lose the race! LOL. Not expecting any different behavior - well, not any better behavior anyway. But hey, 11.4b2 booted to the login screen on the first try, so that's something right?

 

[Macschrauber]

I may have another data point.

Used as the one and only drive an Apple SSBUX AHCI Blade from an MP6,1 I guess (APPLE SSD SM0256G)

Clone of the same System I test all the time with 11.3.1 (20E241).

And even with my "stress test" plugging 5 USB Sticks in - plus Monitor and Keyboard Hubs it survived cold boot.

with oclp 0.1.2 and native with -no_compat_check

 

[startergo]

11.4b3 more or less same as 11.4.b2. From time to time panics:

Spoiler: Panic

panic(cpu 12 caller 0xffffff80037c58f6): Kernel trap at 0xffffff8005ead131, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000000, CR3: 0x0000000007b47000, CR4: 0x00000000000226e0
RAX: 0x0000000000000100, RBX: 0x0000000000000000, RCX: 0x0000000000000000, RDX: 0x0000000000000009
RSP: 0xffffffa1b8c43d80, RBP: 0xffffffa1b8c43db0, RSI: 0x0000000965a3eec7, RDI: 0xffffff93ea2e6e00
R8:  0xffffff93ea2b6cd0, R9:  0xffffff871e2a0300, R10: 0x0000000000000001, R11: 0xffffff93ea26dd80
R12: 0xffffff93ea26dd80, R13: 0xffffff8005eac016, R14: 0x0000000965a3eec7, R15: 0xffffff93ea2e6e00
RFL: 0x0000000000010246, RIP: 0xffffff8005ead131, CS:  0x0000000000000008, SS:  0x0000000000000000
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0xc, PL: 0, VF: 1

Backtrace (CPU 12), Frame : Return Address
0xffffffa1b8c437a0 : 0xffffff800368e0dd mach_kernel : _handle_debugger_trap + 0x3fd
0xffffffa1b8c437f0 : 0xffffff80037d4f33 mach_kernel : _kdp_i386_trap + 0x143
0xffffffa1b8c43830 : 0xffffff80037c552a mach_kernel : _kernel_trap + 0x55a
0xffffffa1b8c43880 : 0xffffff8003632a2f mach_kernel : _return_from_trap + 0xff
0xffffffa1b8c438a0 : 0xffffff800368d8fd mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffa1b8c439c0 : 0xffffff800368dbf3 mach_kernel : _panic_trap_to_debugger + 0x273
0xffffffa1b8c43a30 : 0xffffff8003e9d81a mach_kernel : _panic + 0x54
0xffffffa1b8c43aa0 : 0xffffff80037c58f6 mach_kernel : _sync_iss_to_iks + 0x2c6
0xffffffa1b8c43c20 : 0xffffff80037c55dd mach_kernel : _kernel_trap + 0x60d
0xffffffa1b8c43c70 : 0xffffff8003632a2f mach_kernel : _return_from_trap + 0xff
0xffffffa1b8c43c90 : 0xffffff8005ead131 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController18GetOveragedRequestEy + 0x27
0xffffffa1b8c43db0 : 0xffffff8005eac077 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController15CheckForTimeoutEP18IOTimerEventSource + 0x61
0xffffffa1b8c43de0 : 0xffffff8003df4465 mach_kernel : __ZN18IOTimerEventSource15timeoutSignaledEPvS0_ + 0xa5
0xffffffa1b8c43e50 : 0xffffff8003df4366 mach_kernel : __ZN18IOTimerEventSource17timeoutAndReleaseEPvS0_ + 0xc6
0xffffffa1b8c43e80 : 0xffffff80036d46c5 mach_kernel : _thread_call_delayed_timer + 0x4a5
0xffffffa1b8c43ef0 : 0xffffff80036d5634 mach_kernel : _thread_call_delayed_timer + 0x1414
0xffffffa1b8c43fa0 : 0xffffff800363213e mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         com.apple.iokit.IONVMeFamily(2.1)[43E13757-B839-33F0-9002-5D2F6C945368]@0xffffff8005e9f000->0xffffff8005ec8fff
            dependency: com.apple.driver.AppleEFINVRAM(2.1)[F43510DC-B31C-3896-AC7C-163AE4115EC5]@0xffffff8004b02000->0xffffff8004b0bfff
            dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[291CFF74-B22E-3F71-B4D4-D8BB557973B8]@0xffffff8004cc4000->0xffffff8004cd9fff
            dependency: com.apple.iokit.IOPCIFamily(2.9)[01799EF2-2C4A-3125-991D-AB121767E204]@0xffffff800615d000->0xffffff8006185fff
            dependency: com.apple.iokit.IOReportFamily(47)[93BE7431-4B8C-3D04-83F0-93254ADF8A5E]@0xffffff8006194000->0xffffff8006196fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[A78CAAA2-43A6-38EF-AEDA-3B957D358855]@0xffffff8006260000->0xffffff8006271fff

Process name corresponding to current thread: kernel_task
Boot args: -v -no_compat_check keepsyms=1 debug=0x144 -liludbgall -nvmefaspm

Mac OS version:
Not yet set

Kernel version:
Darwin Kernel Version 20.5.0: Tue May  4 22:32:58 PDT 2021; root:xnu-7195.121.3~4/RELEASE_X86_64
Kernel UUID: 54102A61-CD81-3528-AE3A-0853D9358B51
KernelCache slide: 0x0000000003400000
KernelCache base:  0xffffff8003600000
Kernel slide:      0x0000000003410000
Kernel text base:  0xffffff8003610000
__HIB  text base: 0xffffff8003500000
System model name: MacPro5,1 (Mac-27AD2F918AE68F61)
System shutdown begun: NO
Panic diags file unavailable, panic occurred prior to initialization
Hibernation exit count: 0

System uptime in nanoseconds: 41512237331
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x00000009ace38321
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000025eae1db38 0x0000000000000000

IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleSMC

So I have 8 SATA drives and I either get:

IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleAHCIDiskDriver
IOPlatformPanicAction -> AppleSMC

or:

IOPlatformPanicAction -> AppleSMC

only.
Worth mentioning that if I do SMC reset after the above mentioned panic it boots right away. Also downgraded IOAHCIBlockStorage.kext from 11.2.3 again, but I don't see any effect.

 

[Dayo]

downgraded IOAHCIBlockStorage.kext

One part of the request:

My two main areas of focus at the moment are /S/L/E/AppleSMC.kext and /S/L/E/IOAHCIFamily.kext/Contents/PlugIns/IOAHCIBlockStorage.kext. Those are the two I'd like to try replacing with the ones from 11.2.x, separately and/or together (and possibly including the parent /S/L/E/IOAHCIFamily.kext/ itself).

Lot of work but needs testers to try to close this out methodically.

Would be great if we can focus on this and sort out or see what pops up before picking up other threads.

 

[startergo]

sudo mount -o nobrowse -t apfs  /dev/disk9s3 /System/Volumes/Update/mnt1
Password:
bigsurssd@BigSurSSDs-Mac-Pro ~ % sudo rm -R /System/Volumes/Update/mnt1/System/Library/Extensions/AppleSMC.kext
bigsurssd@BigSurSSDs-Mac-Pro ~ % sudo cp -R /Users/bigsurssd/Desktop/AppleSMC.kext  /System/Volumes/Update/mnt1/System/Library/Extensions/
bigsurssd@BigSurSSDs-Mac-Pro ~ % sudo chmod -Rf 755 /System/Volumes/Update/mnt1/System/Library/Extensions/AppleSMC.kext
bigsurssd@BigSurSSDs-Mac-Pro ~ % sudo chown -Rf root:wheel /System/Volumes/Update/mnt1/System/Library/Extensions/AppleSMC.kext
bigsurssd@BigSurSSDs-Mac-Pro ~ % sudo kmutil install --volume-root /System/Volumes/Update/mnt1/ --update-all
checking collections...
updated extension 'com.promise.driver.stex' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/PromiseSTEX.kext' (mach-o UUID changed from <unknown> to AE253556-66D8-38D2-B9D1-179B78F29153)
updated extension 'com.apple.driver.usb.AppleUSBVHCICommonRSM' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/AppleUSBCommon.kext/Contents/PlugIns/AppleUSBVHCICommonRSM.kext' (mach-o UUID changed from <unknown> to BE3738D0-185A-354D-8386-452EEBAE8A12)
updated extension 'com.Areca.ArcMSR' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/ArcMSR.kext' (mach-o UUID changed from <unknown> to 01FC5EBD-0A34-3E77-9216-53B89DDD8CE4)
updated extension 'com.apple.driver.AppleSMC' found on disk at '/System/Volumes/Update/mnt1/System/Library/Extensions/AppleSMC.kext' (mach-o UUID changed from BDF8B08E-B8C3-30BC-A0ED-727E6AE48479 to 1E48B8EB-7BF3-3366-8520-2C4B31A7D3F8)
rebuilding release collections: boot, system
rebuilding release collections:
    boot kernel collection
    system kext collection
rebuilding local auxiliary collection
kmutil done
bigsurssd@BigSurSSDs-Mac-Pro ~ % sudo bless --folder /System/Volumes/Update/mnt1/System/Library/CoreServices --bootefi --create-snapshot

One part of the request:

Lot of work but needs testers to try to close this out methodically.

You probably missed that I did it with the previous build. I did it now too and it still panicked. Again after SMC reset I boot right away.