Constant Kernel Panics – userspace watchdog timeout: no successful checkins from com.apple.WindowServer

[star-affinity]

I posted here about how I can replicate.

I can now solve this (KP on reboot (Watchdogd))

My replication steps only work when my terminal emulator (iTerm/Terminal) was in added in "Developer Tools" section of Security & Privacy settings in macOS.

Removed them and I could not longer replicate.

This looks to be a bug in codesign/watchdogd/adhoc signatures. Where you remove the terminal from dev tools, you can see the signature check is performed and all is well.

If you have terminal in the "developer tools" category, certain tools (like the ones I describe in my post, ruby/pip), cause the OS to panic on reboot, as something ends up hanging.

Amazing but now solved...

Described further here: https://sigpipe.macromates.com/2020/macos-catalina-slow-by-design/

Fantastic job isolating it like that!

But still curios if that solves the problem for everyone that has it? It seems there can be different causes for ”watchdog timeout”, so I guess not…

 

[random-uk]

@johnalan looks like I spoke too soon re MacPorts - had the same kernel panic on reboot (with no homebrew installed) after three days of the machine being up. So scratch that idea.

I completely removed MacPorts, reinstalled homebrew then only installed python, rebooted and tried your pip3 list test, rebooted and no kernel panic.

Interesting your find on having terminal in developer tools in sys prefs. I will test that now. When you say "you can see the signature check is performed" how are you testing that?

I was also wondering if there might be any process hanging due to iCloud drive... Out of interest, do you have any ln -s to anything in iCloud drive (on the assumption you use it)?

 

[johnalan]

@johnalan looks like I spoke too soon re MacPorts - had the same kernel panic on reboot (with no homebrew installed) after three days of the machine being up. So scratch that idea.

I completely removed MacPorts, reinstalled homebrew then only installed python, rebooted and tried your pip3 list test, rebooted and no kernel panic.

Interesting your find on having terminal in developer tools in sys prefs. I will test that now. When you say "you can see the signature check is performed" how are you testing that?

I was also wondering if there might be any process hanging due to iCloud drive... Out of interest, do you have any ln -s to anything in iCloud drive (on the assumption you use it)?

echo $'#!/bin/sh\necho Hello' > /tmp/test.sh && chmod a+x /tmp/test.sh
time /tmp/test.sh && time /tmp/test.sh

This test shows the signature check... you can see it will have times like this first time:


/tmp/test.sh 0.00s user 0.00s system 1% cpu 0.338 total
/tmp/test.sh 0.00s user 0.00s system 70% cpu 0.006 total

but times like this the second time:


/tmp/test.sh 0.00s user 0.00s system 1% cpu 0.005 total
/tmp/test.sh 0.00s user 0.00s system 70% cpu 0.003 total


It's faster because of the OCSP check is cached.

This is the root of the issue when your emulator is marked as "Developer Tool" i.e. exempt from this check, but a sub-process is running unsigned code.

I think anyway

 

[random-uk]

echo $'#!/bin/sh\necho Hello' > /tmp/test.sh && chmod a+x /tmp/test.sh
time /tmp/test.sh && time /tmp/test.sh

This test shows the signature check... you can see it will have times like this first time:


/tmp/test.sh 0.00s user 0.00s system 1% cpu 0.338 total
/tmp/test.sh 0.00s user 0.00s system 70% cpu 0.006 total

but times like this the second time:


/tmp/test.sh 0.00s user 0.00s system 1% cpu 0.005 total
/tmp/test.sh 0.00s user 0.00s system 70% cpu 0.003 total


It's faster because of the OCSP check is cached.

This is the root of the issue when your emulator is marked as "Developer Tool" i.e. exempt from this check, but a sub-process is running unsigned code.

I think anyway

really interesting... I will run more tests here and see if your approach solves it. God I'd love for this to be the solution!

 

[johnalan]

really interesting... I will run more tests here and see if your approach solves it. God I'd love for this to be the solution!

If your problem was the kernel panic on reboot with watchdogd saying “no checkins” I think it’s likely fixed!

let me know!

 

[random-uk]

Here's one of my many, many kernel panics:

panic

But for me, it normally seems to require the machine being up for 2-3 days and when rebooting after that time to test I get a kernel panic. So I will report back

 

[chrfr]

If your problem was the kernel panic on reboot with watchdogd saying “no checkins” I think it’s likely fixed!

let me know!

I very sporadically get those panics on restart but don't have Macports or Homebrew installed and don't have Terminal enabled as a developer tool in the Security and Privacy preferences. So while those may be a cause of these panics, it's not the only cause.

 

[KevinGrabher]

I am baffled that you'd rather get MLBs and entire machines replaced than to properly troubleshoot the issue.

I have reverse engineered macOS, followed the backtrace and call stack in Xcode, reversed engineered the SMC bin in Ghidra trying to trouble shoot this.
I've already isolated the likely problematic code and spoken with macOS engineering about it, which is part of XNU.

I have a control MBP with vanilla Big Sur 11.1, never had anything but the Apple charger plugged into it, no user-land software outside of stock Apple apps, no kernel extensions, literally out of the box, and it has this issue.


"beyond obvious" top ****ing kek.

Thank you for your sagacious insight, wonderful to finally have a path forward.

@johnalan
Would you agree, in hindsight, that this post of yours aged rather poorly?

 

[johnalan]

Here's one of my many, many kernel panics:

panic

But for me, it normally seems to require the machine being up for 2-3 days and when rebooting after that time to test I get a kernel panic. So I will report back

Yep I think my approach should fix this. Let me know!

 

[johnalan]

@johnalan
Would you agree, in hindsight, that this post of yours aged rather poorly?

No it’s absolutely a problem in XNU. This is just a workaround.

The core logic in the kernel handling exec of OSCP checked binaries panics watchdogd.

Surprised you can’t see that implicitly here.

User-space shouldn’t interfere with Kernel-space

 

[johnalan]

I very sporadically get those panics on restart but don't have Macports or Homebrew installed and don't have Terminal enabled as a developer tool in the Security and Privacy preferences. So while those may be a cause of these panics, it's not the only cause.

I can understand. This happens whenever an elevated process executed a non OSCP check bin. There may be many ways to cause this. It’s a Kernel issue.

thankfully my approach is like flipping a switch for the stuff I use. I can turn panics on and off consistently now.

 

[chrfr]

I can understand. This happens whenever an elevated process executed a non OSCP check bin. There may be many ways to cause this. It’s a Kernel issue.

thankfully my approach is like flipping a switch for the stuff I use. I can turn panics on and off consistently now.

I do hope that you've filed this information with Apple. If it's that consistent for you it might be enough data to actually get this resolved for everyone.

 

[johnalan]

I do hope that you've filed this information with Apple. If it's that consistent for you it might be enough data to actually get this resolved for everyone.

I have provided detailed feedback - updating the case with my findings on Feedback Assistant ( Have FB numbers if interested ) - it's been over 8 months with no feedback.

I gave some stupid feedback on iOS UI years ago and got feedback within a week.

🤷

 

[dretheviking]

If your problem was the kernel panic on reboot with watchdogd saying “no checkins” I think it’s likely fixed!

let me know!

Hi! I'm on BigSur 2013 macpro and been having these frequent watchdogs no checkin crash/reboots after updating to BigSur. I can't find the developer tools section in system preferences security & privacy settings. Would you know if BigSur has this section and if there is a similar fix. Thank you for your time. Andreas

 

[johnalan]

Hi! I'm on BigSur 2013 macpro and been having these frequent watchdogs no checkin crash/reboots after updating to BigSur. I can't find the developer tools section in system preferences security & privacy settings. Would you know if BigSur has this section and if there is a similar fix. Thank you for your time. Andreas

it looks like this in Big Sur, haven't had a panic in over a week for the first time in a year.

 

[random-uk]

it looks like this in Big Sur, haven't had a panic in over a week for the first time in a year.

View attachment 1735887

If you don't have the Developer Tools section, this will make it appear

sudo spctl developer-mode enable-terminal

 

[johnalan]

If you don't have the Developer Tools section, this will make it appear

sudo spctl developer-mode enable-terminal

Have you had the panic since?

 

[MusicDev]

Registered an account here just to say johnalan, I cannot thank you enough. I have been following this thread since it started.

Before this, I had tried everything, removed system & user caches, rebuilt fonts, reinstalled homebrew, reinstalled git , reinstalled/updated basically all my dev tools, reinstalled OSX, upgraded to Big Sur.... everything. Nothing worked.

Have you had the panic since?

I think you've fixed my problem, I removed references to terminal from Security & Privacy in system prefs from developer tools tab & others (belt & braces and all that) and my ~2015 MBP Big Sur hasn't crashed since. Before this, I couldn't use my Mac unplugged from power, or without an external monitor (weirdly it was stable if connected to external screen). Without power & external monitor I was stuck in an endless panic loop, had major display & performance issues. I almost bought a new MacBook thinking this was the end for this machine.

Also another point, even if Terminal/iTerm does not have the checkbox next to it checked, remove it anyway.

To reiterate & expand johnalan's fix:
- Boot into safe mode (Hold shift at power on) if you are in a panic loop
- System preferences -> Security & Privacy
- Remove terminal from Developer Tools section (I also removed it from accessibility & other areas)
- Power off
- Power on

Computer hasn't crashed since, been around 24 hours now, able to use it without external monitor and power supply. Previously without external monitor & power my laptop was useless. I now have my laptop back.

johnalan, & all other contributors, thank you!

 

[random-uk]

Have you had the panic since?

Well for me I've not been able to reproduce the panic like you can by running pip3 list for example. The most reliable way for me to reproduce is to leave the machine up for 3 days then try a restart. I'm presumably doing something in my day to day programming that is triggering it but I've got no idea what. So I waited three days since your breakthrough and just restarted now. Kernel panic. But a different kind - i.e. not a WatchDog timeout error.

New panic

So just for the sake of things I zapped my PRAM and did an SMC reset. I seriously doubt that will do anything but at least it's done. Now I'll wait another 3 days and see what happens. Joy. If that different kind of kernel panic suggests anything to try I would love any thoughts.

 

[johnalan]

Registered an account here just to say johnalan, I cannot thank you enough. I have been following this thread since it started.

Before this, I had tried everything, removed system & user caches, rebuilt fonts, reinstalled homebrew, reinstalled git , reinstalled/updated basically all my dev tools, reinstalled OSX, upgraded to Big Sur.... everything. Nothing worked.



I think you've fixed my problem, I removed references to terminal from Security & Privacy in system prefs from developer tools tab & others (belt & braces and all that) and my ~2015 MBP Big Sur hasn't crashed since. Before this, I couldn't use my Mac unplugged from power, or without an external monitor (weirdly it was stable if connected to external screen). Without power & external monitor I was stuck in an endless panic loop, had major display & performance issues. I almost bought a new MacBook thinking this was the end for this machine.

Also another point, even if Terminal/iTerm does not have the checkbox next to it checked, remove it anyway.

To reiterate & expand johnalan's fix:
- Boot into safe mode (Hold shift at power on) if you are in a panic loop
- System preferences -> Security & Privacy
- Remove terminal from Developer Tools section (I also removed it from accessibility & other areas)
- Power off
- Power on

Computer hasn't crashed since, been around 24 hours now, able to use it without external monitor and power supply. Previously without external monitor & power my laptop was useless. I now have my laptop back.

johnalan, & all other contributors, thank you!

Thanks for your thoughtful note!

Delighted to hear I've helped you!

 

[johnalan]

Well for me I've not been able to reproduce the panic like you can by running pip3 list for example. The most reliable way for me to reproduce is to leave the machine up for 3 days then try a restart. I'm presumably doing something in my day to day programming that is triggering it but I've got no idea what. So I waited three days since your breakthrough and just restarted now. Kernel panic. But a different kind - i.e. not a WatchDog timeout error.

New panic

So just for the sake of things I zapped my PRAM and did an SMC reset. I seriously doubt that will do anything but at least it's done. Now I'll wait another 3 days and see what happens. Joy. If that different kind of kernel panic suggests anything to try I would love any thoughts.

hmmm intersting, I've never seen this panic, I've only had watchdogd, which is now solved a week later, still none my end.


Might be an anomaly? you were having watchdogd too right?

 

[random-uk]

hmmm intersting, I've never seen this panic, I've only had watchdogd, which is now solved a week later, still none my end.


Might be an anomaly? you were having watchdogd too right?

Oh yes I was having the exact watchdogd panics... So, so many watchdogd panics. Like a misbehaved dog craving attention

Fingers crossed for an anomaly. I will post when I have further results!

 

[gregorskii]

I posted here about how I can replicate.

I can now solve this (KP on reboot (Watchdogd))

My replication steps only work when my terminal emulator (iTerm/Terminal) was in added in "Developer Tools" section of Security & Privacy settings in macOS.

Removed them and I could not longer replicate.

This looks to be a bug in codesign/watchdogd/adhoc signatures. Where you remove the terminal from dev tools, you can see the signature check is performed and all is well.

If you have terminal in the "developer tools" category, certain tools (like the ones I describe in my post, ruby/pip), cause the OS to panic on reboot, as something ends up hanging.

Amazing but now solved...

Described further here: https://sigpipe.macromates.com/2020/macos-catalina-slow-by-design/

Interesting, mine shows there but it's not checked.

I do not have the issues on reboot, but I do not reboot my mac that often.

 

[MusicDev]

Hi all,

Hope you can help me. I've installed Big Sur 11.2.2 in update from 11 on an external USB HD connected via an adapter on a 2019 16" MacBook Pro and keep getting crashes every so often, whereby the mouse pointer disappears and the system becomes unresponsive and either runs the fans and reboots and says the system rebooted blah... or I have to hold the power button and do it manually.

I have spoken to Apple and they are useless!

Crash log attached from recent crash. UAudio and Waves Soundgrid plugins/apps/drivers are all up-to-date and compatible with Big Sur as you'll see the kext note.

Thanks in advance for any help, I don't have developer tools in security and privacy and don't run hombrew or ruby or other such detailed in past replies.

Ash

Hi Ash,

If it is not present for you, I wouldn't worry, Kernel panics can be caused by a lot of stuff but since I am a fellow music producer as well I will try my best to help you!

You could try:
- Open terminal
- Type sudo nvram -c
- Enter your password (it will not appear but don't worry the input is working)
- Hit enter
- There will possibly be some error but don't worry
- Restart your machine

Another thing, I removed my UA Audio software recently (sold my Apollo) and found that my system was much more stable. As for Waves I have found them not to be an issue for system stability.

Another thing to note is if you are running your OS on an external HD is not going to be as fast as using the internal SSD and I would probably switch to using the internal one.

 

[MusicDev]

Cool job! I'm a web developer by trade but I write music in my downtime.

The bit in the panic log about kexts is just telling you what is loaded, it probably isn't related to the panic.
The panic is obviously caused by the no checkins from watchdog (the reason we're all here) which has dependency extensions as you can read in the output below the backtrace.

My deeply uneducated guess is this can be caused by a bunch of things as we are slowly unraveling in this thread every use-case is slightly different.

The only reason I mentioned about the UA plugins was when I uninstalled the UA software using the official UA uninstaller my machine then prompted me about a bunch of outdated extensions which I assume were related to UA. Although, if you are up to date it's unlikely to be the issue (I hadn't used mine in about a year).

One other thing you can try as it seems from your log there are some boot arguments. If you didn't write/enable those specifically you could try:

sudo nvram -d boot-args

It's clear from the length and breadth of the content of this thread that this watchdog issue is multifaceted. Considering they market these products as 'Pro' which implies they are designed to be professional tools, I can't believe the lack of support on this.

EDIT (to better answer with some more ideas):
- Delete Cache files from ~/Library/Caches (WARNING if you use a DAW it'll have to scan all of your plugins again, and you'll probs lose other system/software config settings)
- Try a safe mode boot (hold shift at apple logo) then just restart
- Reinstall OS X (you can do it without losing data, obviously a crappy option)