MP 1,1-5,1 cMP 4,1 (5,1) crashing after CPU upgrade

[sgentile92]

That should suffice, although I sometimes wait for 4th chime.

Run that terminal command. You can change 10m to 5h or whatever time frame you need. Curious to see what SMC is reporting as the crash reason.

Hmm...this doesn't sound great:

Admins-Mac-Pro:~ admintest$ log show --predicate 'eventMessage contains "Previous shutdown cause"' --last 50m
log: warning: The log archive contains partial or missing metadata
log: cannot use --last when archive metadata is missing

 

[amstel78]

Oof. Sounds like data corruption going on there.

Just a wild thought, but is your cMP plugged into any sort of surge protector or directly into the wall? Have you tried plugging directly into another outlet?

I ask because years ago, I had a Windows box do something kind of similar, except only during heavy gaming or running GPU benchmarks with two Nvidia cards in an SLI setup. Drove me crazy for a month trying to figure out why it kept crashing. Even resorted to swapping out the power supply thinking the PS couldn't supply enough power to the cards.

Turns out that the outlet it was plugged into suffered from voltage sag. It wasn't enough to trip a breaker in the panel but voltage and amps dipped low enough that it triggered a system reset. Moving the computer to another outlet solved the issue. Go figure.

 

[sgentile92]

It had been plugged into a wall outlet but a couple hours ago I plugged into a surge protector connected to another outlet. No change.

Any idea how to fix that data corruption so the terminal command will work? Or why it happened?

 

[amstel78]

You could try running Onyx to clear log files and caches, but if your OS is getting corrupted due to memory or other issues, not sure if it'll help. You may need to create a clean install on known good Mac then transfer the drive over.

I'll have to defer to the more knowledgeable folks here moving forward.

Edit: have a look at this

Jim DeLaHunt, world-ready » Resolving “log archive metadata is missing” and “error number 65”

blog.jdlh.com

 

[tsialex]

Do you have the Panic Log when you log back after the crashes?

 

[sgentile92]

Do you have the Panic Log when you log back after the crashes?

Below is one of the reports generated since installing the new RAM. Note: this particular report isn't from a crash while running a clean install of OS—let me know if that would be easier to decipher, but the same problem occurs on the clean install drive too.

I kept a number of these over the last couple of years dealing with this problem (before I caved and reinstalled my pair of 2.26s, which have since been upgraded, sending me down this rabbit hole once more) and the "9 error-reporting banks" and processors 12–23 being listed is always consistent—even when I used a different set of 3.33s—but I can't figure out what that means. I've spent a bunch of time trying to figure how to decipher it but it still doesn't make sense to me.

Spoiler: Panic Log

Anonymous UUID: 58FA0CE6-8281-4D56-07DD-43EEEC2ACD26

Tue Mar 9 14:09:00 2021

*** Panic Report ***
Machine-check capabilities: 0x0000000000001c09
family: 6 model: 44 stepping: 2 microcode: 31
signature: 0x206c2
Intel(R) Xeon(R) CPU X5680 @ 3.33GHz
9 error-reporting banks
Processor 12: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 13: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 14: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 15: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 16: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 17: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 18: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 19: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 20: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 21: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 22: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 23: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
mp_kdp_enter(): 16777151, 1, 24 TIMED-OUT WAITING FOR NMI-ACK, PROCEEDING
panic(cpu 6 caller 0xffffff80199871da): "Machine Check at 0x00007fff64850745, registers:\n" "CR0: 0x0000000080010033, CR2: 0x000060c000278000, CR3: 0x00000016b8fe50f4, CR4: 0x00000000000226e0\n" "RAX: 0x0000000000000000, RBX: 0x000060c000278000, RCX: 0x0000000000010000, RDX: 0x0000000000000507\n" "RSP: 0x00007ffeee875e48, RBP: 0x00007ffeee875e60, RSI: 0x0000000000000507, RDI: 0x00007fff964c6c48\n" "R8: 0x0000000000000004, R9: 0x0000000000000001, R10: 0x0000000000000000, R11: 0x0000000000000000\n" "R12: 0x003fffffffffffff, R13: 0xffc00fffffffffff, R14: 0x00007fff3c4cde70, R15: 0x0000001004004010\n" "RFL: 0x0000000000000246, RIP: 0x00007fff64850745, CS: 0x000000000000002b, SS: 0x0000000000000023\n" "Error code: 0x0000000000000000\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.71.82.6/osfmk/i386/trap_native.c:168
Backtrace (CPU 6), Frame : Return Address
0xffffffbac73c3c00 : 0xffffff801986ae56
0xffffffbac73c3c50 : 0xffffff8019994434
0xffffffbac73c3c90 : 0xffffff8019986604
0xffffffbac73c3d00 : 0xffffff801981ce60
0xffffffbac73c3d20 : 0xffffff801986a8cc
0xffffffbac73c3e50 : 0xffffff801986a68c
0xffffffbac73c3eb0 : 0xffffff80199871da
0xffffffbac73c3fa0 : 0xffffff801981d6df

BSD process name corresponding to current thread: Spotlight

Mac OS version:
17G14033

Kernel version:
Darwin Kernel Version 17.7.0: Mon Aug 31 22:11:23 PDT 2020; root:xnu-4570.71.82.6~1/RELEASE_X86_64
Kernel UUID: 92BEC910-BBAA-3192-BB57-39712C7D3342
Kernel slide: 0x0000000019600000
Kernel text base: 0xffffff8019800000
__HIB text base: 0xffffff8019700000
System model name: MacPro5,1 (Mac-F221BEC8)

System uptime in nanoseconds: 46215370798
last loaded kext at 27708468659: com.apple.driver.AppleHIDKeyboard 205.1 (addr 0xffffff7f9c47b000, size 45056)
loaded kexts:
com.nvidia.CUDA 1.1.0
com.blackmagic-design.desktopvideo.iokit.framebufferdriver 10.4.1
com.newer-tech.kext.nwtmem 1.0.0
com.apple.driver.AudioAUUC 1.70
com.apple.driver.AppleTyMCEDriver 1.0.3d2
com.apple.driver.AGPM 110.23.37
com.apple.filesystems.autofs 3.0
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.driver.AppleMikeyHIDDriver 131
com.apple.driver.AppleGraphicsDevicePolicy 3.20.18
com.apple.AGDCPluginDisplayMetrics 3.20.18
com.apple.driver.AppleHDAHardwareConfigDriver 281.52
com.apple.driver.AppleHDA 281.52
com.apple.driver.AppleMikeyDriver 281.52
com.apple.driver.AppleUpstreamUserClient 3.6.5
com.apple.driver.AppleHV 1
com.apple.driver.AppleMCCSControl 1.5.5
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.kext.AMDLegacyFramebuffer 1.6.8
com.apple.iokit.IOBluetoothSerialManager 6.0.7f22
com.apple.driver.pmtelemetry 1
com.apple.ATIRadeonX2000 10.0.4
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.kext.AMD4800Controller 1.6.8
com.apple.driver.AppleLPC 3.1
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleFIVRDriver 4.1.0
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.iokit.IOBluetoothUSBDFU 6.0.7f22
com.apple.filesystems.apfs 748.51.0
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.filesystems.hfs.kext 407.50.6
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleSystemPolicy 1.0
com.apple.iokit.SCSITaskUserClient 404.30.3
com.apple.driver.AppleFWOHCI 5.5.9
com.apple.driver.Intel82574LEthernet 2.7.2
com.apple.driver.AirPort.Brcm4331 800.21.30
com.apple.driver.AppleAHCIPort 329.50.2
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleACPIButtons 6.1
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleIntelCPUPowerManagementClient 220.50.1
com.apple.driver.AppleACPIEC 6.1
com.apple.driver.AppleAPIC 1.7
com.apple.nke.applicationfirewall 186
com.apple.security.TMSafetyNet 8
com.apple.security.quarantine 3
com.apple.driver.AppleIntelCPUPowerManagement 220.50.1
com.apple.driver.AppleHIDKeyboard 205.1
com.apple.driver.IOBluetoothHIDDriver 6.0.7f22
com.apple.kext.triggers 1.0
com.apple.driver.AppleGraphicsControl 3.20.18
com.apple.driver.DspFuncLib 281.52
com.apple.kext.OSvKernDSPLib 526
com.apple.iokit.IOAVBFamily 683.1
com.apple.plugin.IOgPTPPlugin 680.15
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.AppleSSE 1.0
com.apple.iokit.IOSurface 211.15
com.apple.iokit.IOSerialFamily 11
com.apple.iokit.IONDRVSupport 519.21
com.apple.AppleGPUWrangler 3.20.18
com.apple.driver.AppleSMBusController 1.0.18d1
com.apple.kext.AMDSupport 1.6.8
com.apple.AppleGraphicsDeviceControl 3.20.18
com.apple.driver.AppleHDAController 281.52
com.apple.iokit.IOHDAFamily 281.52
com.apple.iokit.IOAudioFamily 206.5
com.apple.vecLib.kext 1.2.0
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.kext.AMDLegacySupport 1.6.8
com.apple.iokit.IOGraphicsFamily 519.23
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOFireWireIP 2.2.9
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 6.0.7f22
com.apple.iokit.IOBluetoothHostControllerUSBTransport 6.0.7f22
com.apple.iokit.IOBluetoothHostControllerTransport 6.0.7f22
com.apple.iokit.IOBluetoothFamily 6.0.7f22
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.AppleXsanScheme 3
com.apple.iokit.IOAHCIBlockStorage 301.40.2
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.filesystems.hfs.encodings.kext 1
com.apple.iokit.IOSCSIMultimediaCommandsDevice 404.30.3
com.apple.iokit.IOBDStorageFamily 1.8
com.apple.iokit.IODVDStorageFamily 1.8
com.apple.iokit.IOCDStorageFamily 1.8
com.apple.iokit.IOAHCISerialATAPI 267.50.1
com.apple.iokit.IOFireWireFamily 4.7.2
com.apple.iokit.IO80211Family 1200.12.2
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleUSBMergeNub 900.4.1
com.apple.iokit.IOAHCIFamily 288
com.apple.driver.usb.AppleUSBEHCIPCI 1.2
com.apple.driver.usb.AppleUSBUHCIPCI 1.2
com.apple.driver.usb.AppleUSBUHCI 1.2
com.apple.driver.usb.AppleUSBEHCI 1.2
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOUSBFamily 900.4.1
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.AppleEFINVRAM 2.1
com.apple.driver.AppleEFIRuntime 2.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.DiskImages 480.60.3
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleUSBTDM 439.70.3
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.iokit.IOUSBMassStorageDriver 140.70.2
com.apple.iokit.IOSCSIBlockCommandsDevice 404.30.3
com.apple.iokit.IOSCSIArchitectureModelFamily 404.30.3
com.apple.iokit.IOStorageFamily 2.1
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleBusPowerController 1.0
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.IOSlaveProcessor 1
com.apple.iokit.IOReportFamily 31
com.apple.iokit.IOTimeSyncFamily 680.15
com.apple.iokit.IONetworkingFamily 3.4
com.apple.driver.AppleACPIPlatform 6.1
com.apple.driver.AppleSMC 3.1.9
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 1.0

EOF
Model: MacPro5,1, BootROM 144.0.0.0.0, 12 processors, 6-Core Intel Xeon, 3.33 GHz, 64 GB, SMC 1.39f5
Graphics: ATI Radeon HD 4870, ATI Radeon HD 4870, PCIe
Memory Module: DIMM 1, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
Memory Module: DIMM 2, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
Memory Module: DIMM 5, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
Memory Module: DIMM 6, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x8E), Broadcom BCM43xx 1.0 (5.106.98.102.30)
Bluetooth: Version 6.0.7f22, 3 services, 27 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en2
PCI Card: ATI Radeon HD 4870, Display Controller, Slot-1
Serial ATA Device: HL-DT-ST DVD-RW GH41N
Serial ATA Device: OWC Mercury EXTREME Pro 6G SSD, 120.03 GB
Serial ATA Device: TOSHIBA MD04ACA400, 4 TB
USB Device: USB Bus
USB Device: USB Bus
USB Device: BRCM2046 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Optical Mouse
USB Device: USB Bus
USB Device: Hub in Apple Pro Keyboard
USB Device: Apple Pro Keyboard
USB Device: USB 2.0 Bus
USB Device: USB 2.0 Bus
FireWire Device: built-in_hub, Up to 800 Mb/sec
Thunderbolt Bus:

 

[tsialex]

Below is one of the reports generated since installing the new RAM. Note: this particular report isn't from a crash while running a clean install of OS—let me know if that would be easier to decipher, but the same problem occurs on the clean install drive too.

I kept a number of these over the last couple of years dealing with this problem (before I caved and reinstalled my pair of 2.26s, which have since been upgraded, sending me down this rabbit hole once more) and the "9 error-reporting banks" and processors 12–23 being listed is always consistent—even when I used a different set of 3.33s—but I can't figure out what that means. I've spent a bunch of time trying to figure how to decipher it but it still doesn't make sense to me.

Spoiler: Panic Log

Anonymous UUID: 58FA0CE6-8281-4D56-07DD-43EEEC2ACD26

Tue Mar 9 14:09:00 2021

*** Panic Report ***
Machine-check capabilities: 0x0000000000001c09
family: 6 model: 44 stepping: 2 microcode: 31
signature: 0x206c2
Intel(R) Xeon(R) CPU X5680 @ 3.33GHz
9 error-reporting banks
Processor 12: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 13: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 14: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 15: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 16: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 17: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 18: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 19: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 20: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 21: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 22: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
Processor 23: IA32_MCG_STATUS: 0x0000000000000004
IA32_MC6_STATUS(0x419): 0xbe00000000300186
IA32_MC6_ADDR(0x41a): 0x00000017d39388c0
IA32_MC6_MISC(0x41b): 0x0000000080003032
mp_kdp_enter(): 16777151, 1, 24 TIMED-OUT WAITING FOR NMI-ACK, PROCEEDING
panic(cpu 6 caller 0xffffff80199871da): "Machine Check at 0x00007fff64850745, registers:\n" "CR0: 0x0000000080010033, CR2: 0x000060c000278000, CR3: 0x00000016b8fe50f4, CR4: 0x00000000000226e0\n" "RAX: 0x0000000000000000, RBX: 0x000060c000278000, RCX: 0x0000000000010000, RDX: 0x0000000000000507\n" "RSP: 0x00007ffeee875e48, RBP: 0x00007ffeee875e60, RSI: 0x0000000000000507, RDI: 0x00007fff964c6c48\n" "R8: 0x0000000000000004, R9: 0x0000000000000001, R10: 0x0000000000000000, R11: 0x0000000000000000\n" "R12: 0x003fffffffffffff, R13: 0xffc00fffffffffff, R14: 0x00007fff3c4cde70, R15: 0x0000001004004010\n" "RFL: 0x0000000000000246, RIP: 0x00007fff64850745, CS: 0x000000000000002b, SS: 0x0000000000000023\n" "Error code: 0x0000000000000000\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.71.82.6/osfmk/i386/trap_native.c:168
Backtrace (CPU 6), Frame : Return Address
0xffffffbac73c3c00 : 0xffffff801986ae56
0xffffffbac73c3c50 : 0xffffff8019994434
0xffffffbac73c3c90 : 0xffffff8019986604
0xffffffbac73c3d00 : 0xffffff801981ce60
0xffffffbac73c3d20 : 0xffffff801986a8cc
0xffffffbac73c3e50 : 0xffffff801986a68c
0xffffffbac73c3eb0 : 0xffffff80199871da
0xffffffbac73c3fa0 : 0xffffff801981d6df

BSD process name corresponding to current thread: Spotlight

Mac OS version:
17G14033

Kernel version:
Darwin Kernel Version 17.7.0: Mon Aug 31 22:11:23 PDT 2020; root:xnu-4570.71.82.6~1/RELEASE_X86_64
Kernel UUID: 92BEC910-BBAA-3192-BB57-39712C7D3342
Kernel slide: 0x0000000019600000
Kernel text base: 0xffffff8019800000
__HIB text base: 0xffffff8019700000
System model name: MacPro5,1 (Mac-F221BEC8)

System uptime in nanoseconds: 46215370798
last loaded kext at 27708468659: com.apple.driver.AppleHIDKeyboard 205.1 (addr 0xffffff7f9c47b000, size 45056)
loaded kexts:
com.nvidia.CUDA 1.1.0
com.blackmagic-design.desktopvideo.iokit.framebufferdriver 10.4.1
com.newer-tech.kext.nwtmem 1.0.0
com.apple.driver.AudioAUUC 1.70
com.apple.driver.AppleTyMCEDriver 1.0.3d2
com.apple.driver.AGPM 110.23.37
com.apple.filesystems.autofs 3.0
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.driver.AppleMikeyHIDDriver 131
com.apple.driver.AppleGraphicsDevicePolicy 3.20.18
com.apple.AGDCPluginDisplayMetrics 3.20.18
com.apple.driver.AppleHDAHardwareConfigDriver 281.52
com.apple.driver.AppleHDA 281.52
com.apple.driver.AppleMikeyDriver 281.52
com.apple.driver.AppleUpstreamUserClient 3.6.5
com.apple.driver.AppleHV 1
com.apple.driver.AppleMCCSControl 1.5.5
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.kext.AMDLegacyFramebuffer 1.6.8
com.apple.iokit.IOBluetoothSerialManager 6.0.7f22
com.apple.driver.pmtelemetry 1
com.apple.ATIRadeonX2000 10.0.4
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.kext.AMD4800Controller 1.6.8
com.apple.driver.AppleLPC 3.1
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleFIVRDriver 4.1.0
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.iokit.IOBluetoothUSBDFU 6.0.7f22
com.apple.filesystems.apfs 748.51.0
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.filesystems.hfs.kext 407.50.6
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleSystemPolicy 1.0
com.apple.iokit.SCSITaskUserClient 404.30.3
com.apple.driver.AppleFWOHCI 5.5.9
com.apple.driver.Intel82574LEthernet 2.7.2
com.apple.driver.AirPort.Brcm4331 800.21.30
com.apple.driver.AppleAHCIPort 329.50.2
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleACPIButtons 6.1
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleIntelCPUPowerManagementClient 220.50.1
com.apple.driver.AppleACPIEC 6.1
com.apple.driver.AppleAPIC 1.7
com.apple.nke.applicationfirewall 186
com.apple.security.TMSafetyNet 8
com.apple.security.quarantine 3
com.apple.driver.AppleIntelCPUPowerManagement 220.50.1
com.apple.driver.AppleHIDKeyboard 205.1
com.apple.driver.IOBluetoothHIDDriver 6.0.7f22
com.apple.kext.triggers 1.0
com.apple.driver.AppleGraphicsControl 3.20.18
com.apple.driver.DspFuncLib 281.52
com.apple.kext.OSvKernDSPLib 526
com.apple.iokit.IOAVBFamily 683.1
com.apple.plugin.IOgPTPPlugin 680.15
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.AppleSSE 1.0
com.apple.iokit.IOSurface 211.15
com.apple.iokit.IOSerialFamily 11
com.apple.iokit.IONDRVSupport 519.21
com.apple.AppleGPUWrangler 3.20.18
com.apple.driver.AppleSMBusController 1.0.18d1
com.apple.kext.AMDSupport 1.6.8
com.apple.AppleGraphicsDeviceControl 3.20.18
com.apple.driver.AppleHDAController 281.52
com.apple.iokit.IOHDAFamily 281.52
com.apple.iokit.IOAudioFamily 206.5
com.apple.vecLib.kext 1.2.0
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.kext.AMDLegacySupport 1.6.8
com.apple.iokit.IOGraphicsFamily 519.23
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOFireWireIP 2.2.9
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 6.0.7f22
com.apple.iokit.IOBluetoothHostControllerUSBTransport 6.0.7f22
com.apple.iokit.IOBluetoothHostControllerTransport 6.0.7f22
com.apple.iokit.IOBluetoothFamily 6.0.7f22
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.AppleXsanScheme 3
com.apple.iokit.IOAHCIBlockStorage 301.40.2
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.filesystems.hfs.encodings.kext 1
com.apple.iokit.IOSCSIMultimediaCommandsDevice 404.30.3
com.apple.iokit.IOBDStorageFamily 1.8
com.apple.iokit.IODVDStorageFamily 1.8
com.apple.iokit.IOCDStorageFamily 1.8
com.apple.iokit.IOAHCISerialATAPI 267.50.1
com.apple.iokit.IOFireWireFamily 4.7.2
com.apple.iokit.IO80211Family 1200.12.2
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleUSBMergeNub 900.4.1
com.apple.iokit.IOAHCIFamily 288
com.apple.driver.usb.AppleUSBEHCIPCI 1.2
com.apple.driver.usb.AppleUSBUHCIPCI 1.2
com.apple.driver.usb.AppleUSBUHCI 1.2
com.apple.driver.usb.AppleUSBEHCI 1.2
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOUSBFamily 900.4.1
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.AppleEFINVRAM 2.1
com.apple.driver.AppleEFIRuntime 2.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.DiskImages 480.60.3
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleUSBTDM 439.70.3
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.iokit.IOUSBMassStorageDriver 140.70.2
com.apple.iokit.IOSCSIBlockCommandsDevice 404.30.3
com.apple.iokit.IOSCSIArchitectureModelFamily 404.30.3
com.apple.iokit.IOStorageFamily 2.1
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleBusPowerController 1.0
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.IOSlaveProcessor 1
com.apple.iokit.IOReportFamily 31
com.apple.iokit.IOTimeSyncFamily 680.15
com.apple.iokit.IONetworkingFamily 3.4
com.apple.driver.AppleACPIPlatform 6.1
com.apple.driver.AppleSMC 3.1.9
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 1.0

EOF
Model: MacPro5,1, BootROM 144.0.0.0.0, 12 processors, 6-Core Intel Xeon, 3.33 GHz, 64 GB, SMC 1.39f5
Graphics: ATI Radeon HD 4870, ATI Radeon HD 4870, PCIe
Memory Module: DIMM 1, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
Memory Module: DIMM 2, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
Memory Module: DIMM 5, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
Memory Module: DIMM 6, 16 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342324737304442302D594B302020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x8E), Broadcom BCM43xx 1.0 (5.106.98.102.30)
Bluetooth: Version 6.0.7f22, 3 services, 27 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en2
PCI Card: ATI Radeon HD 4870, Display Controller, Slot-1
Serial ATA Device: HL-DT-ST DVD-RW GH41N
Serial ATA Device: OWC Mercury EXTREME Pro 6G SSD, 120.03 GB
Serial ATA Device: TOSHIBA MD04ACA400, 4 TB
USB Device: USB Bus
USB Device: USB Bus
USB Device: BRCM2046 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Optical Mouse
USB Device: USB Bus
USB Device: Hub in Apple Pro Keyboard
USB Device: Apple Pro Keyboard
USB Device: USB 2.0 Bus
USB Device: USB 2.0 Bus
FireWire Device: built-in_hub, Up to 800 Mb/sec
Thunderbolt Bus:

This is the important part 24 TIMED-OUT WAITING FOR NMI-ACK, NMI are non-maskable interrupts.

You probably have a problem with your CPU tray QPI link that makes NUMA work, since you already tried other processor pairs, could be a CPU tray itself problem.

Do you have another tray to test your Mac Pro?

 

[sgentile92]

This is the important part 24 TIMED-OUT WAITING FOR NMI-ACK, NMI are non-maskable interrupts.

You probably have a problem with your CPU tray QPI link that makes NUMA work, since you already tried other processor pairs, could be a CPU tray itself problem.

Do you have another tray to test your Mac Pro?

I don't, which is very frustrating. Been searching for a good deal on a dual CPU 4,1 to try it out in. Any thoughts on why everything would be stable with the original 2.26s? Is a faulty CPU board something that Apple Hardware or Diagnostic test can pick up on?

Also, probable shot in the dark: I just plugged my Mac Pro into an energy meter and during high stress it draws upwards of 400w plus. Is that normal? I don't recall where I read it but I thought I heard that during stress it's only supposed to draw somewhere in the 300w range.

 

[tsialex]

I don't, which is very frustrating. Been searching for a good deal on a dual CPU 4,1 to try it out in. Any thoughts on why everything would be stable with the original 2.26s? Is a faulty CPU board something that Apple Hardware or Diagnostic test can pick up on?

Some early dual CPU trays don't work well with upgraded Xeons, some people think it's the 130W CPUs, others think that is the 1333MHz QPI link.

ASD should confirm the NMI/QPI problem. Since it's a MP4,1>5,1, you should use mid-2010 ASD.

Also, probable shot in the dark: I just plugged my Mac Pro into an energy meter and during high stress it draws upwards of 400w plus. Is that normal? I don't recall where I read it but I thought I heard that during stress it's only supposed to draw somewhere in the 300w range.

It's not unheard, but wouldn't be a surprise that you have a problematic PSU.

This type of problem is more easily solved with a working Mac Pro to test each component of the problematic one.

Btw, check the neutral/phase pinout of your outlets, a lot of places have it wired incorrectly.

 

[sgentile92]

Some early dual CPU trays don't work well with upgraded Xeons, some people think it's the 130W CPUs, others think that is the 1333MHz QPI link.

ASD should confirm the NMI/QPI problem. Since it's a MP4,1>5,1, you should use mid-2010 ASD.


It's not unheard, but wouldn't be a surprise that you have a problematic PSU.

This type of problem is more easily solved with a working Mac Pro to test each component of the problematic one.

Btw, check the neutral/phase pinout of your outlets, a lot of places have it wired incorrectly.

I ran ASD for mid-2010 Mac Pro, which takes about an hour or so for the hardware test. My problem is that twice now, it's either restarted during the final leg of the memory test or after completing the test—I'm not sure because i wasn't watching it at that exact moment. Is it typical to restart after completing the test or does it sound like it's crashing during that portion of the test?

I'm weighing my options here and deciding the best course of action. Seems like I could go with another 2009 and swap trays and whatnot, OR if some early 2009 dual CPU trays are problematic with upgraded Xeons, would it perhaps make more sense at this point to look for a 2010–2012 5,1, sell off both pairs of delidded 3.33s and buy the cheaper lidded processors that the 2010-2012 models require and install my RAM and GTX 970 in that? Is there enough of an advantage to using a genuine 5,1 to warrant that?

 

[tsialex]

I ran ASD for mid-2010 Mac Pro, which takes about an hour or so for the hardware test. My problem is that twice now, it's either restarted during the final leg of the memory test or after completing the test—I'm not sure because i wasn't watching it at that exact moment. Is it typical to restart after completing the test or does it sound like it's crashing during that portion of the test?

During the test.

I'm weighing my options here and deciding the best course of action. Seems like I could go with another 2009 and swap trays and whatnot, OR if some early 2009 dual CPU trays are problematic with upgraded Xeons, would it perhaps make more sense at this point to look for a 2010–2012 5,1, sell off both pairs of delidded 3.33s and buy the cheaper lidded processors that the 2010-2012 models require and install my RAM and GTX 970 in that? Is there enough of an advantage to using a genuine 5,1 to warrant that?

It's a very low percentage, but if you use the search you will find some cases.

You need to go for what makes more sense to you, maybe it's cheaper to go for an early-2009 bought locally, but a mid-2012 is three years newer.

A mid-2010/mid-2012 essentially just have different EFI and SMC firmware versions after you remove GPU/HDD/CPU. No performance gains whatsoever, but the SMC don't have the racing fan bug.

 

[sgentile92]

During the test.


It's a very low percentage, but if you use the search you will find some cases.

You need to go for what makes more sense to you, maybe it's cheaper to go for an early-2009 bought locally, but a mid-2012 is three years newer.

A mid-2010/mid-2012 essentially just have different EFI and SMC firmware versions after you remove GPU/HDD/CPU. No performance gains whatsoever, but the SMC don't have the racing fan bug.

Ok, so playing detective, restarting during the memory portion of the test while running a new set of RAM would seem to back the theory of a problem with the CPU tray, correct? Or does the 400W + draw during stress also potentially indicate a potential problem with the PSU? If I possibly have two faulty parts that could definitely impact the decision I make.

Thanks for taking the time to help me though this, I really appreciate it.

 

[tsialex]

Ok, so playing detective, restarting during the memory portion of the test while running a new set of RAM would seem to back the theory of a problem with the CPU tray, correct? Or does the 400W + draw during stress also potentially indicate a potential problem with the PSU? If I possibly have two faulty parts that could definitely impact the decision I make.

Thanks for taking the time to help me though this, I really appreciate it.

Both could be problematic at the same time - Mac Pro design a lot of times is robust enough that takes more than one part not working perfectly for a hard crash. Defective/dying PSUs usually still "work" when you are not pushing hard.

The best way to solve this is getting a similar and confirmed working Mac Pro to test each component. I'd start installing a the defective PSU in a know working Mac Pro and a working PSU in the defective one.

 

[Macsonic]

Thanks for taking the time to help me though this, I really appreciate it.

If you have not done this, after the chime sound, try checking if there is a red light coming out of your CPU tray if the X5680s are installed. This might provide some clue on the source of the problem and the guys here can give their inputs. Though, tsialex’s suggestion on getting another working 2009 Mac Pro and test out each component is one of the best option I could think of.

 

[panjandrum]

Sounds like a tough process-of-elimination (excuse me, *component isolation*) issue for sure. Just going to chime in here quickly to say that I've upgraded a total of 24 4,1s flashed to 5,1s in an educational setting, with anywhere from dual 5690s (3.46GHz) downward; none of them de-lidded (used the washer-method). The 3 fastest of these machines I use personally for doing things like handbrake conversions and running Stream to Me media servers (which often converts media on the fly) and the like, so I peg the processors near 100% in 3 of these boxes for extended periods of time on a routine basis. Zero problems with power supplies or temps (I *DO* run Macs Fan Control on them to keep temps a bit cooler than they otherwise would be). The rest of the machines are used by students with zero reported issues, including some fun stuff like the Universe Sandbox, which push the CPUs and GPUs pretty hard and thus draw considerable power (AMD 560s, 570s and 580s for video cards).

I can report that two of these three fastest machines are at 3.46GHz, and one of them is at 3.33GHz. The 3.33GHz machine is less reliable. It's "fine", but I end up restarting that once at least once every couple of weeks. The other two, at 3.46HGz are rock-solid for months on end. For whatever that's worth.

So all else being equal, what you are trying to do should be very reliable in the end on your 2009 machine. At least, it's certainly been great for me across a pretty large sample-size.

I admit to not having read the entire thread post-by-post, but if I were you I would do these two steps next if you can possibly arrange it. It sounds like you've already tried a different set of de-lidded processors and swapped out RAM? 1) swap the processors out for a pair of lidded (hence cheaper) processors at a different speed just in case there is something weird at 3.33GHz on some machines (see my comment above, might as well go to 3.46GHz IMHO) 2) swap out the power supply 3) swap out the Nvidia video card for an AMD video card. Sure, neither the CPU frequency nor the Video card should be the cause of the problems, but at this point you almost have to test for those. There isn't much else I can think of at this point. Past those items and you are down to either the motherboard or the cpu tray.

 

[Macsonic]

I can report that two of these three fastest machines are at 3.46GHz, and one of them is at 3.33GHz. The 3.33GHz machine is less reliable. It's "fine", but I end up restarting that once at least once every couple of weeks. The other two, at 3.46HGz are rock-solid for months on end. For whatever that's worth.

So all else being equal, what you are trying to do should be very reliable in the end on your 2009 machine. At least, it's certainly been great for me across a pretty large sample-size.

Hi Panjandrum. Just to clarify, you mentioned the Mac Pro with 3.33Ghz x5680 cpus was less reliable. Would you advise that the x5680s cpus are to be avoided on 2009 Mac Pros? Thanks

 

[panjandrum]

Hi Panjandrum. Just to clarify, you mentioned the Mac Pro with 3.33Ghz x5680 cpus was less reliable. Would you advise that the x5680s cpus are to be avoided on 2009 Mac Pros? Thanks

My sample size is far too small to know an answer, as that's my *only* one that has the x5680 in it; the rest of my 24 upgraded Mac Pros have other variants. It is the only one that seems even slightly flaky of the bunch, but that could be any number of things and it's plenty stable enough that I've never bothered trying to figure it out. Sorry.

Now, if sgentile92 comes back and says he replaced his 3.33GHz chips with a different speed and *bang* it suddenly worked fine, then we've got a sample size of 2. Better than 1!

 

[Macsonic]

My sample size is far too small to know an answer, as that's my *only* one that has the x5680 in it; the rest of my 24 upgraded Mac Pros have other variants. It is the only one that seems even slightly flaky of the bunch, but that could be any number of things and it's plenty stable enough that I've never bothered trying to figure it out. Sorry.

Now, if sgentile92 comes back and says he replaced his 3.33GHz chips with a different speed and *bang* it suddenly worked fine, then we've got a sample size of 2. Better than 1!

Thanks for the added info. In the past I upgraded a couple of 4,1, 2009s Mac Pros though the cpus that were used were delidded x5675 and x5670. Both worked fine. Hope sgentile92 finds a solution Thanks

 

[tsialex]

3.33GHz (W3680/X5680) it's not the problem at all - all my Mac Pros have one or two 3.33GHz hexa, none have this problem @panjandrum.

My early-2009 20H B08 works perfectly with dual X5680s with uptimes of months. The reliability problem happens with the early runs of the dual CPU tray, the later revision works perfectly.

 

[rdalrt]

Just throwing this out there because I had a similar issue with my 4,1 crashing during benchmarking, etc.

Where did you get the delidded processors? Do they have any scratches/nicks whatsoever on the pcb? One pair of xeons I delidded myself acted flakey after reinstalling them. Turns out on closer inspection, on one of the processors there was one tiny little scratch on the pcb out beside where the adhesive on the lid was. After replacing that processor with another, all was good.

 

[Cfreak]

Ok, so playing detective, restarting during the memory portion of the test while running a new set of RAM would seem to back the theory of a problem with the CPU tray, correct? Or does the 400W + draw during stress also potentially indicate a potential problem with the PSU? If I possibly have two faulty parts that could definitely impact the decision I make.

Thanks for taking the time to help me though this, I really appreciate it.

Did you ever get tis solved? I have the same processors and the same crashing issue, with the same "9 error-reporting banks" message, but less frequent and often when waking from sleep or opening a youtube page.