MP 7,1 Random Panic/Sleep Issues Persist

[Theophilos]

Hello,

Some time ago, I reported issues that I had with a Highpoint SSD 7101 card with four Samsung NVME drives (2 x 1 TB, 2 x2 TB) installed. When this card was installed in Slot 5, my Mac Pro panicked and crashed when sleeping. I was disappointed at the HP card, because I also have a Sonnet 4x4 with 4 Samsung NVME drives (all 1TB) installed in Slot 4 with no issues.

I contacted Apple Support and we engaged in troubleshooting for about two months. They had me remove the card, remove my third-party RAM, try each SSD, etc., all while asking me send detailed dumps of my logs and system files, which I did. At the end of this painstaking process, Apple's engineers pointed at the High Point card and said it was the problem.

So I returned the High Point card and ordered another Sonnet 4x4, which just came yesterday. Sure enough, after installing it in Slot 5, the panic while sleeping issue returned.

I am at a loss what to do. What do you think this is? Any suggestions would be appreciated.

FYI:

- All Samsung drives are Evo Plus with the latest firmware.

- Mac Pro is running the latest release of Catalina, not beta software.

- We already confirmed the issue exists when using the original RAM, so it’s probably not my third-party 384 GB.

- I know I could just disable sleep, but I don’t want to, because it doesn’t seem right to waste power needlessly in this day and age. Also, for $12,000, I expect a computer that can execute basic functions, like sleep and wake up peacefully.

 

[Schismz]

TL;DR: disable all "power saving/waste hours of your time instead?" "features."

1. Catalina 10.15.4 in particular, is kinda a POS. 10.15.3 was very solid for me; 10.15.4 has introduced multiple random annoying problems, but anyhoo, life goes on, hopefully 10.15.5 will work a bit better.

2. Sleep... IDK. I've just turned my workstations on and then let them run forever 24/7 365 since the Apple ][ days. I sleep the displays, and disable all other power saving features and experience far fewer problems.

If it's a portable device, I get it. If it's a MP, why do you care? Within the context of how much you spent on the hardware, saving $80 a year in electricity or whatever it is where you live, matters?

Having said that, fully agreed, it *should* Just Work, but, alas, it often doesn't circa Catalina. Dear Apple, my time has value; I'm not getting paid to be your technician/assistant engineer and spend hours collaborating with you on debugging your OS software for you. For the prices you charge, Make it Go without annoying me and wasting my time. -- But, they haven't, and it doesn't, so... turn off energy saving features until they work correctly (should be no later than 2025 or so).

p.s., "Mac Pro is running the latest release of Catalina, not beta software" Hahahahahahahahaha, uhm, ya. Keep telling yourself that, we're all beta testers. Go launch Sidecar and experience true happiness at the magnificent progress and magical features you can enjoy instead.

...

Being pragmatic, and setting aside what is vs. what should be, try setting your Energy Saver like this, and see if it resolves your problems.

 

[Theophilos]

I get it, this is all a huge waste of my time, but again, we are living in a new age of global threats. My conscience would not be at peace if I knew I was needlessly wasting energy (even if I could afford it). Apple touts its environmental impact in every product announcement. It seems to me the functionality to save energy on every desktop Mac and display by gracefully sleeping/waking would make a significant impact on the environment. This needs to be fixed.

 

[Snow Tiger]

I contacted Apple Support and we engaged in troubleshooting for about two months .

Never a good indication . ?‍♂️

 

[OkiRun]

TL;DR: disable all "power saving/waste hours of your time instead?" "features."

1. Catalina 10.15.4 in particular, is kinda a POS. 10.15.3 was very solid for me; 10.15.4 has introduced multiple random annoying problems, but anyhoo, life goes on, hopefully 10.15.5 will work a bit better.

2. Sleep... IDK. I've just turned my workstations on and then let them run forever 24/7 365 since the Apple ][ days. I sleep the displays -- or turned them off using a button <gasp> in the olden days -- and disable all other power saving features and experience far fewer issues and dumb problems.

If it's a portable device, I get it. If it's a MP, why do you care? Within the context of how much you spent on the hardware, saving $80 a year in electricity or whatever it is where you live, matters?

Having said that, fully agreed, it *should* Just Work, but, alas, it often doesn't circa Catalina. Dear Apple, my time has value, as should be obvious from how much I've spent on your hardware, I'm not getting paid to be your technician/assistant engineer and debug software for you. For the prices you charge, Make it Go without annoying me and wasting my time. -- But, they haven't, and it doesn't, so... turn off energy saving features until they work right (should be no later than 2025 or so).

p.s., "Mac Pro is running the latest release of Catalina, not beta software" Hahahahahahahahaha, uhm, ya. Keep telling yourself that, we're all beta testers. Go launch Sidecar and experience true happiness at the magnificent progress and magical features you can enjoy instead.

...

Being pragmatic, and setting aside what is vs. what should be, try setting your Energy Saver like this, and see if it stops your problems.

View attachment 912683

I use the same settings and run the 7.1 24/7 as well. No issues. In exchange ~ I recycle everything possible instead of sending it to the dump, drive a small gas efficient car, and help clean the local beach on weekends with like minded friends. I sleep well even if my 7.1 doesn't

 

[xeux]

Thanks for sharing your experience. I can't figure out why my 7,1 basically locks up after waking from sleep. I'm on 10.15.4 and it's pretty consistent. I just tried disabling sleep, and hope it resolves it until 10.15.5.

 

[Bakafish]

I disabled sleep and haven't had any issues since then. It likely has nothing to do with your 3rd party stuff (mine is stock.) I'm happy to pay a few yen wasting power and helping accelerate the Earth's demise a few milliseconds just to avoid dealing with my Mac not being remotely accessible when I need it and having to worry about lost work. Plus my office is a little warmer in the mornings now, bonus.

 

[xeux]

Mine is stock too. Anyways, disabling sleep fixed my issue it seems.

 

[Aboo]

Same here - had multiple issues and like you OP, I too changed out the highpoint card for the sonnet, but was still experiencing issues. I finally disabled the sleep features as advised here and things seem more stable now. I also upgraded to the 10.15.5 beta 4 because I got the W5700X card, and this seems to have helped too.

 

[th0masp]

Sleep seems like a pretty basic function these days, if it can't handle that - what else may be broken between macOS and drivers for thirdparty hardware, just not as easily exposed?

Btw. have you tried to change the system sleep mode for hibernation (via pmset in Terminal) - any difference?

 

[handheldgames]

Sleep seems like a pretty basic function these days, if it can't handle that - what else may be broken between macOS and drivers for thirdparty hardware, just not as easily exposed?

Btw. have you tried to change the system sleep mode for hibernation (via pmset in Terminal) - any difference?

That's third party hardware, on what is essentially a reference design from broadcom using drivers from apple, that just so happens to run flawless on older macs and pc's. Considering the target market and the slots, NVMe expansion cards are a given. We could blame the OS, but it's more of a 7,1 issue.

For the price... It -->should<-- work without issue.

 

[Theophilos]

Hello,

FWIW, Apple contacted me today after their engineers reviewed my logs. They said my 7,1 has a logic board failure, which causes the sleep issue. Since it is covered by AppleCare, we are working on scheduling a repair now.

Even though it has taken months, I am glad to finally have a concrete answer.

 

[Bakafish]

Not to "Well actually" you, but sleep functionality in modern Intel systems is actually very complex, with a whole lot of different states and different parts of the CPU and support chipsets turning on and off functionality to a running OS. This has been complicated by Apple's use of the T2 processor in several modern models, which has its own hooks deep into these operations (where it seems a lot of the issues are coming from.)

Part of the issue that Apple seems to be having is that when these subsystems are in their hibernation states and something goes wrong, there's not a lot of ability to log what is going on and the two different 'brains' of the system are likely at odds over what to do about it. For the first few months my box couldn't even generate a crash dump report when this happened. The later crash reports I saw didn't look all that much more informative to my layman's eye, and without replication and good tracing it is hard to fix these bugs. So I'll agree with you that this should work, but I'm not surprised that they are having issues.

<CONSPIRACY THEORY/>
My (completely unsubstantiated) guess is that the T2 integration at such a low level on the new Mac Pro was done for much more than security (as Intel's TPM solution could handle all of that years ago) but as a way to transition to an ARM based processor that can plug into one of the PCI or MPX slots. This meant that they had to give the T2 all sorts of low level access and control over the Intel CPU, memory and PCI systems that will eventually pay off by arbitrating between the two architectures during simultaneous operation, but currently is creating a lot of stability issues around bringing the Intel side up and down. My feeling is that they will continue to sell the Mac Pro in its current form for a long time, leveraging the Intel architecture for rock solid backwards compatibility and performance for pro users while allowing ARM based applications running natively at the same time. But maybe that's just my mind trying to rationalize the massive chunk of change I dropped on an Intel rig in the dying days of its reign.

 

[Schismz]

That's third party hardware, on what is essentially a reference design from broadcom using drivers from apple, that just so happens to run flawless on older macs and pc's. Considering the target market and the slots, NVMe expansion cards are a given. We could blame the OS, but it's more of a 7,1 issue.

For the price... It -->should<-- work without issue.

I kinda think it's a T2 issue, not a 7,1 issue... If you look through the forums here, the sleep problem seems to be affecting multiple systems, all of which have one thing in common ... they're all on the T2 chip.

The T2 was my entire concern before the 7,1 ever shipped, because the iMac Pro I dumped had the same stupid problems with sleep and power management. Now it works! Now it doesn't. Now it kernel panics, now ... it's time to turn off all power management/sleep features, and magically the problems all go away.

POLL: When to buy a revised cheese grater Mac Pro?

Buying one right away if not sooner... Although it's obviously infinitely flawed in a wide variety of ways which I can never get over, none-the-less, it is CHEESEGRATER! I

forums.macrumors.com

And that's on Mojave which overall was far more stable than Catalina...

Not to "Well actually" you, but sleep functionality in modern Intel systems is actually very complex, with a whole lot of different states and different parts of the CPU and support chipsets turning on and off functionality to a running OS. This has been complicated by Apple's use of the T2 processor in several modern models, which has its own hooks deep into these operations (where it seems a lot of the issues are coming from.)

Same opinion. It's the T2.

Hello,

FWIW, Apple contacted me today after their engineers reviewed my logs. They said my 7,1 has a logic board failure, which causes the sleep issue. Since it is covered by AppleCare, we are working on scheduling a repair now.

Even though it has taken months, I am glad to finally have a concrete answer.

I feel you. I hope it resolves your sleep issues, and perhaps you do have a logic board problem. But... these are the same people who told you the problem was your Highpoint card... so, maybe they just feel compelled to do something for you. I'm sure Apple does care, and I'm sure they're trying to fix the problem, but...

<CONSPIRACY THEORY/>
My (completely unsubstantiated) guess is that the T2 integration at such a low level on the new Mac Pro was done for much more than security (as Intel's TPM solution could handle all of that years ago) but as a way to transition to an ARM based processor that can plug into one of the PCI or MPX slots. This meant that they had to give the T2 all sorts of low level access and control over the Intel CPU, memory and PCI systems that will eventually pay off by arbitrating between the two architectures during simultaneous operation, but currently is creating a lot of stability issues around bringing the Intel side up and down. My feeling is that they will continue to sell the Mac Pro in its current form for a long time, leveraging the Intel architecture for rock solid backwards compatibility and performance for pro users while allowing ARM based applications running natively at the same time. But maybe that's just my mind trying to rationalize the massive chunk of change I dropped on an Intel rig in the dying days of its reign.

I want to believe. Please pass the Koolaid, will start holding breath

<Gazing into Crystal Ball> 2023: Apple proudly unveils T3 chip, "power management works correctly now!" version.

 

[daveedjackson]

in the last 2 days I've had 4 or 6 panics. The latest just now. System is idle. No software open or running. system not doing anything at all. yet come to it "system unexpectedly shutdown" it's becoming more frequent, and less acceptable.

Any thoughts?

panic(cpu 0 caller 0xffffff8002816487): "PRT0::setPowerState(0xffffff815e632000 : 0xffffff7f84a1663a, 0 -> 2) timed out after 30369 ms"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.101.6/iokit/Kernel/IOServicePM.cpp:5296
Backtrace (CPU 0), Frame : Return Address
0xffffff92e667bb40 : 0xffffff80021215cd
0xffffff92e667bb90 : 0xffffff800225a3c5
0xffffff92e667bbd0 : 0xffffff800224bf7e
0xffffff92e667bc20 : 0xffffff80020c7a40
0xffffff92e667bc40 : 0xffffff8002120c97
0xffffff92e667bd40 : 0xffffff8002121087
0xffffff92e667bd90 : 0xffffff80028c2c7c
0xffffff92e667be00 : 0xffffff8002816487
0xffffff92e667be50 : 0xffffff8002815d69
0xffffff92e667be60 : 0xffffff800282d2fe
0xffffff92e667bea0 : 0xffffff8002814b18
0xffffff92e667bec0 : 0xffffff8002163545
0xffffff92e667bf40 : 0xffffff8002163071
0xffffff92e667bfa0 : 0xffffff80020c713e

BSD process name corresponding to current thread: kernel_task
Boot args: chunklist-security-epoch=0 -chunklist-no-rev2-dev

Mac OS version:
19E287

Kernel version:
Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64
Kernel UUID: AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6
Kernel slide: 0x0000000001e00000
Kernel text base: 0xffffff8002000000
__HIB text base: 0xffffff8001f00000
System model name: MacPro7,1 (Mac-27AD2F918AE68F61)
System shutdown begun: NO

System uptime in nanoseconds: 20996480046077
last loaded kext at 37690351110: com.getdropbox.dropbox.kext 1.11.0 (addr 0xffffff7f890de000, size 49152)
last unloaded kext at 17075688044628: |IOAHCIBlock!S 316.100.5 (addr 0xffffff7f857f2000, size 163840)
loaded kexts:
com.getdropbox.dropbox.kext 1.11.0
>AudioAUUC 1.70
>!AUpstreamUserClient 3.6.8

 

[Oval]

Investigating & Preventing Kernel Panics in the 2019 Mac Pro

Nobody likes to experience a kernel panic. This is when your Mac restarts itself while you are working, and you may lose that work if your files are not saved.

blog.macsales.com

 

[Theophilos]

Investigating & Preventing Kernel Panics in the 2019 Mac Pro

Nobody likes to experience a kernel panic. This is when your Mac restarts itself while you are working, and you may lose that work if your files are not saved.

blog.macsales.com

Thanks. I wish I had seen this before I let Apple pick up my Mac. This is exactly the issue I have.

 

[Theophilos]

I received my Mac Pro back from repair today. They replaced the logic board, but the panic while sleep issue persists. Clearly, it is a MacOS bug that Apple’s engineers are not admitting publicly.

The link from Oval above seems to describe the real cause and solution for this problem. I followed the guidance in the link and the problem is fixed temporarily. Thanks to the OWC engineers for their amazing troubleshooting skills and thanks to Oval for finding and sharing this.

 

[daveedjackson]

I received my Mac Pro back from repair today. They replaced the logic board, but the panic while sleep issue persists. Clearly, it is a MacOS bug that Apple’s engineers are not admitting publicly.

The link from Oval above seems to describe the real cause and solution for this problem. I followed the guidance in the link and the problem is fixed temporarily. Thanks to the OWC engineers for their amazing troubleshooting skills and thanks to Oval for finding and sharing this.

The it ticket posted is about having the OWC card installed. Which I do not. So how does this best address the panics? I don’t get it. Also why would Apple replace the logic board because of panics. When obviously many of us have had this issue.

 

[th0masp]

Some nice detective work in that article. Would be pretty bizarre if Apple did not realise where the problem was or the engineers did not communicate it to the repair staff and went and replaced the mainboard instead.

 

[eflx]

The ticket isn't specifically about an OWC card if you read again. It's just that in that case, an OWC card was used so the company (OWC) decided to look into it further, and found that it was potentially some sort of timing issue when high-bandwidth/low latency PCIe cards were installed in those slots, and set to use Pool B of the PCIe lanes.

 

[Aboo]

Interesting - their article specifically mentions the Highpoint Card as causing this issue. @Theopilos - I recall you having similar issues with your High Point card no?

EDIT: Just saw your original post; thought this was a separate thread for some reason. It at least looks like Apple Engineering is aware of this issue. I am sure OWC has a relationship with Apple Engineering and thus can get their attention more easily than us mere mortals. Hopefully this means that a solution is on the way.

 

[MarciaFunebre]

... Clearly, it is a MacOS bug that Apple’s engineers are not admitting publicly.
...

I think there are a lot of those.
Today a new Beta Version of MacOS was released. I wonder if it addresses any of the issues. Of course, and thank you Apple, no documentation whatsoever!

 

[handheldgames]

I think there are a lot of those.
Today a new Beta Version of MacOS was released. I wonder if it addresses any of the issues. Of course, and thank you Apple, no documentation whatsoever!

Can this be fixed with an OS update or a Motherboard revision? Apple's T2 is a rats nest of issues and has been for years. Long live T2!!

 

[Theophilos]

I am very sorry to report that today's 10.15.5 update did not solve this issue, at least not for my config:

Mac Pro (2019)
16c
384 GB RAM
2 x Sonnet m.2 4x4 loaded with Samsung Evo Plus drives in Slots 4 and 5

Disappointing to say the least.