Mac Pro (Late 2013) GPU (Driver) Issues

[Levv]

UPDATE: Big Sur much better as Catalina, but still freezes 1-3 times per day

Is there an improvement with Big Sur 11.2 RC2 or haven't you downloaded it yet?

 

[Robo_G]

After install Big Sur 11.2 newest beta (20D62) there is no any improvements... maybe even worse (MP 3 times freezes today)

 

[Levv]

After install Big Sur 11.2 newest beta (20D62) there is no any improvements... maybe even worse (MP 3 times freezes today)

Hmmm, I'll stay on Mojave.... Why don't you downgrade? Mojave is pretty stable on my Mp.

 

[b_Michaeł]

After install Big Sur 11.2 newest beta (20D62) there is no any improvements... maybe even worse (MP 3 times freezes today)

Hi.

What firmware version do you have for your graphics card?
What's the date?
Below is a screenshot from the Sensei app.
I have had a Mac Pro for a year. For the first six months, I worked on Mojave. For the next six months - Catalina.
I currently have the latest version of Catalina 10.15.7.
During this time, I have not noticed the problems you describe.

 

[Robo_G]

Hi.

What firmware version do you have for your graphics card?
What's the date?
Below is a screenshot from the Sensei app.
I have had a Mac Pro for a year. For the first six months, I worked on Mojave. For the next six months - Catalina.
I currently have the latest version of Catalina 10.15.7.
During this time, I have not noticed the problems you describe. View attachment 1720831

Hi, I've exactly the same parameters on my MPro...

 

[Robo_G]

Hmmm, I'll stay on Mojave.... Why don't you downgrade? Mojave is pretty stable on my Mp.

I want use the newest operating system and the newest functions on my old MPro ;-)
but maybe I'll downgrade to Mojave

 

[randomtask72]

We need someone brave to let us know when it is safe to upgrade ;-)

 

[bigp2]

I don’t think it’ll ever be safe to upgrade. I can’t see Apple putting any effort into fixing an issue with the low end gpu on an almost obsolete Max Pro.

 

[b_Michaeł]

I want use the newest operating system and the newest functions on my old MPro ;-)
but maybe I'll downgrade to Mojave

Everyone here praises Mojave. Personally, I have a completely different opinion on this subject.
In my opinion, Catalina works much better. Very smooth and responsive.
In addition, I noticed that Mojave used much more GPU power while browsing the web or displaying the graphical interface of applications such as Logic Pro or Ableton Live.

 

[Levv]

Everyone here praises Mojave. Personally, I have a completely different opinion on this subject.
In my opinion, Catalina works much better. Very smooth and responsive.
In addition, I noticed that Mojave used much more GPU power while browsing the web or displaying the graphical interface of applications such as Logic Pro or Ableton Live.

Catalina causes my mac to crash at least 3 times a day (screen freeze). With mojave this happens 3 times a month in a bad month.
So yes, mojave is much better for me, as much as I would love to use the newest version.

 

[eepyikes]

Hi.

What firmware version do you have for your graphics card?
What's the date?
Below is a screenshot from the Sensei app.
I have had a Mac Pro for a year. For the first six months, I worked on Mojave. For the next six months - Catalina.
I currently have the latest version of Catalina 10.15.7.
During this time, I have not noticed the problems you describe. View attachment 1720831

Hey b_Michael, I have the same firmware version (this can also be viewed in About this Mac>>system Report)...while I haven't tried Catalina yet, I had the crashes with every OSX version except Mojave. So yours doesn't crash at all?

 

[b_Michaeł]

Hi. Here is my latest system crash. As you can see, it doesn't apply to GPU problems. After this crash, I uninstalled google chrome and so far I have not noticed any problem.

 

[netkas]

Hi. Here is my latest system crash. As you can see, it doesn't apply to GPU problems. After this crash, I uninstalled google chrome and so far I have not noticed any problem.

you have the usual (for nmp) WindowServer process freeze and then say it's not gpu problems ?

 

[axantas]

As Chrome is in the game here: New behaviour in Bootcamp - crashing the MP 2013 100%:
Use Chrome, goto ntv.de, watch a video - or try to watch it, you will not succeed. There is no immediate crash. Windows is still working, kind of, but dies step by step.

 

[b_Michaeł]

you have the usual (for nmp) WindowServer process freeze and then say it's not gpu problems ?

Most users at the beginning of this topic describe GPu issues as: Kernel_20xx-xx-xx-032655_Mac-Pro.gpuRestart.
So far, I have not encountered this type of failure personally.
In addition, the failure that I presented is more widely described in this topic: userspace watchdog timeout

 

[jeyf]

MB performance has been speratic through the years. definitly hit or miss what you end up with. Apple, a 40+ year old company, is inserting medium quality hardware into a premium market hoping the grand parents will buy it for the demon spawn

 

[incumbent]

I've started bugging Apple about this. They currently have me:

- Use a different account and see if it panics with the same gpuRestart issue
- If so, boot to safe mode and see if it happens

The safe mode one is going to suck, although when I boot it in safe mode it doesn't load the X4000 extension. It does load AMD7000Controller though. Regular boot logged in as my separate admin ID `sudo kextstat | grep -i AMD` says:

  119    2 0xffffff7f97dcc000 0xcf000    0xcf000    com.apple.kext.AMDSupport (4.0.4) 4844AB64-007C-338B-9AD6-736E28D5E928 <118 117 14 13 8 6 5 3 1>
  120    0 0xffffff7f80e55000 0x53000    0x53000    com.apple.kext.AMD7000Controller (4.0.4) 4270E8FC-4DC1-30B3-A77F-2A3296B3513E <119 118 117 14 13 6 5 3 1>
  156    0 0xffffff7f814b6000 0x3000     0x3000     com.apple.kext.AMDRadeonX4000HWServices (4.0.4) 5A7A88AE-66BF-3460-B3EC-138728BBE32F <117 14 13 8 6 5 3 1>
  158    0 0xffffff7f81030000 0x434000   0x434000   com.apple.kext.AMDRadeonX4000 (4.0.4) 66EF4738-AC2F-3B17-8DCB-906F706CCD2C <157 117 90 14 8 6 5 3 1>
  159    0 0xffffff7f8261b000 0xf5000    0xf5000    com.apple.kext.AMDRadeonX4030HWLibs (1.0) DC9C47B9-0455-3750-92D1-68A3222840C6 <14 6 5 3 1>
  160    0 0xffffff7f80fff000 0x28000    0x28000    com.apple.kext.AMDFramebuffer (4.0.4) 97FB00CC-9FFC-3EE1-B0BC-CABB007669E8 <119 117 14 13 8 6 5 3 1>
  166    0 0xffffff7f8102b000 0x3000     0x3000     com.apple.kext.AMDRadeonServiceManager (4.0.4) 14E77794-9CC5-381E-A0CE-661079803FD9 <14 5 3 1>

If anyone wants to look at some diagnostic data, here are the 41 GPU crashes that have happened THIS MORNING:

`https://gofile.me/48sEW/N8oXCQ0eP`

That link expires May 28th, but I can make these all day long no problem.

No, I don't expect Apple to do anything about it other than hear from me regularly about it. I hope it stays up the rest of the day as my admin user because it's extremely frustrating trying to use this workstation in safe boot. Plus, my storage pool isn't available in safe boot so I'll basically just login to plex.tv and hit shuffle?! I can't figure out a way to make it spontaneously do it.

Well, that's not entirely true, it will panic if I plug in my USB cable that goes into a CyberPower UPS, but I didn't remember macOS supports many UPSes without third party software and I installed the CyberPower PowerWhatever app (which I run on a 3,1 just fine) and when I plugged in the UPS it kernel panic'ed and so I removed the Cyberpower software entirely from the workstation but it still crashes when I plug in that USB cable. I'll see if it happens right now after I post this.

Edit: it didn't immediately happen. I wonder if it would when I'm logged in as myself though? I might have two problems.

 

[incumbent]

Well this is disappointing, since I stopped using my regular user account there have not been any panics, not even a `.gpuRestart` event. Usually I get 3-4 GPU crashes and then watchdog bounces the computer. I'm going to login to my AD/kerberos ID and start bringing my favorite software back in small batches and see if I can isolate it that way. I still think it's a software/driver issue of course, but it may be one of my stupid little knobs like iStat Menus or something else I don't actually need that computer doing.

edit: omg the UPS is still plugged in and hasn't rebooted :grimace:

 

[uller6]

Well this is disappointing, since I stopped using my regular user account there have not been any panics, not even a `.gpuRestart` event. Usually I get 3-4 GPU crashes and then watchdog bounces the computer. I'm going to login to my AD/kerberos ID and start bringing my favorite software back in small batches and see if I can isolate it that way. I still think it's a software/driver issue of course, but it may be one of my stupid little knobs like iStat Menus or something else I don't actually need that computer doing.

edit: omg the UPS is still plugged in and hasn't rebooted :grimace:

FYI I had trouble with VMware causing these crashes, but always suspected the issue was some combination of software + hardware and not software alone.

 

[julian1]

I recently upgraded my old Mac Pro to the 12 core processor, 128GB of RAM, added a new SSD boot drive, installed latest Big Sur now updated to 11.3.1. It's a nice spec but the machine is crashing quite often with all of the usual symptoms - loss of control but audio continues. The crash reports when they appear (for some reason not always) usually have the line:

Assertion failed: (isReady - AMD: AMD transaction hang or work overload (AMD IOFB)

Below in the fuller log you will also see:

Call to GetPixelInformation() with pixel aperture 0xaa failed with code 0xe00002e6
Call to GetPixelInformation() with pixel aperture 0xbb failed with code 0xe00002e6
IOAF Debugging Info:Accelerator:
fModelName: 'AMD FirePro D300'

So looks clear one of the culprits is one of the AMD D300s (any idea which one btw?). I think the machine had the problem before I upgraded because I remember it crashed sometimes before although maybe not as often. I've been reading about adding eGPU. Has anyone had any success in mitigating this problem using an eGPU? My concern is that Mac Pro may not fully switch over to using the eGPU. I've read that Final Cut Pro, for example, will often continue to make use of the D300s. Any insights into this? An alternative is that I see it is possible to replace the D300s yourself if you can find replacements maybe on eBay. But I feel wary of doing this as they really aren't cheap and may be just as problematic. eGPU seems more promising perhaps? Thoughts?

Crash log

Process: WindowServer [147]
Path: /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/Resources/WindowServer
Identifier: WindowServer
Version: 600.00 (585)
Code Type: X86-64 (Native)
Parent Process: launchd [1]
Responsible: WindowServer [147]
User ID: 88

Date/Time: 2021-05-20 21:06:19.480 -0700
OS Version: macOS 11.3.1 (20E241)
Report Version: 12
Anonymous UUID: 7598D6B3-2781-202F-7EB0-7CAC58121B1D

Sleep/Wake UUID: 086FBB1F-884C-4D06-A777-40B408C8D2CB

Time Awake Since Boot: 4600 seconds

System Integrity Protection: enabled

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Application Specific Information:
StartTime:2021-05-20 19:49:11
GPU:AMD
MetalDevice for accelerator(0x672b): 0x7fbde1e266c8 (MTLDevice: 0x7fbde80d0000)
MetalDevice for accelerator(0x6f03): 0x7fbde3710f88 (MTLDevice: 0x7fbda8008000)
IOService:/AppleACPIPlatformExpert/PCI0@0/AppleACPIPCI/GFXB@3/IOPP/GFX2@0/ATY,MaguroS@3/AMDFramebufferSI
Assertion failed: (isReady - AMD: AMD transaction hang or work overload (AMD IOFB)
DisplayID: 0x4248597
Surface Use Counts: 2(1) 3(3) 11(1)
FB RegID: 4294969082, On Glass SurfaceIDs: 2 11 3, Transactions: [ Active: SurfaceID: 2 ] [ GPU Busy: SurfaceID: 11 ] [ GPU Busy: SurfaceID: 3 ], Notified IsActive SurfaceIDs: 2, 3, 2, Notified IsActive TransactionIDs: 7953, 7954, 7955, TransactionNotificationCode: 0
Call to GetPixelInformation() with pixel aperture 0xaa failed with code 0xe00002e6
Call to GetPixelInformation() with pixel aperture 0xbb failed with code 0xe00002e6
IOAF Debugging Info:Accelerator:
fModelName: 'AMD FirePro D300'
fDeviceTerminated: NO
fGPURestartCount: 14
Event Machine:
fHWErrorStampIdx: [0x010]
Busy channels:
[index]: fGPUStamp | sstamp | Label and status
[0x00f]: 0x00016cec | 0x000170a5 | VMPT is processing stamp 93421 [0x16ced] submitted from 'kernel_task'(0) on Unknown context; 952 pending
[0x00c]: 0x00001f13 | 0x00001f15 | DisplayPipe3 is idle
[0x004]: 0x00005744 | 0x000057a1 | DMA1 is processing stamp 22341 [0x5745] submitted from 'Final Cut Pro'(396) on Unknown context; 92 pending; blocked on stamp 93425 [0x16cf1] on channel 0x00f (VMPT)
[0x003]: 0x00014105 | 0x00014170 | DMA0 is processing stamp 82182 [0x14106] submitted from 'Final Cut Pro'(396) on Unknown context; 106 pending; blocked on stamp 93422 [0x16cee] on channel 0x00f (VMPT)
[0x002]: 0x000086fd | 0x000086fe | Compute1 is processing stamp 34558 [0x86fe] submitted from 'ScreensharingAge'(427) on CL context; blocked on stamp 82192 [0x14110] on channel 0x003 (DMA0); blocked on stamp 126971 [0x1effb] on channel 0x000 (GFX); blocked on stamp 93430 [0x16cf6] on channel 0x00f (VMPT)
[0x000]: 0x0001eff5 | 0x0001f057 | GFX is processing stamp 126966 [0x1eff6] submitted from 'Final Cut Pro'(396) on Metal context; 97 pending; blocked on stamp 82184 [0x14108] on channel 0x003 (DMA0); blocked on stamp 93422 [0x16cee] on channel 0x00f (VMPT)
Events Timing:
FB will power off: 4669688 ms ago
FB did power on: 4669688 ms ago
System Will Sleep: 4669688 ms ago
System Did Wake: 4669688 ms ago
Mode Will change: 62828 ms ago
Mode Did change: 62799 ms ago
Queue state:
0 : ch [0x000] stamp [0x1f055];
1 w : ch [0x000] stamp [0x1efef];
2 L: ch [0x000] stamp [0x1eff4];
3 r : ch [0x000] stamp [0x1effa];
DisplayPipe: TQRC: 7955; TQWC: 7957; Pending: N; Live: Y; Next: 7957; EvtIR: ON; VBLIR: OFF; TransactIR: OFF; NTQID-RC: 7953-0x0 7954-0x0 7955-0x0
STAT e00 MODE 0 SLAV 0 WSAA 1 WSAAPS 1 DISP 0 PWR 2 MODE 80001000 HA 1680 VA 1050 HS 0 VS 0 PCLK 119000 PD 3 DPCH 0 DPSM 0 LHPT 0 STAT e7 PWR 0 0 V Counter diff 48 10a07 0 SMU 1Notified TIDs: 7953 7954 7955 MachPort: 0xb127
IOP Notified Transaction IDs: 7950, 7951, 7952 7953 7954 7955; IOP TransactionCode: 0 IOP TSCH: (7952, 0), (7953, 0), (7954, 0), (7955, 0), (7956, 0), (7957, 0)
), function CoreDisplay_NotReady, file /System/Volumes/Data/SWE/macOS/BuildRoots/2288acc43c/Library/Caches/com.apple.xbs/Sources/CoreDisplay/CoreDisplay-236.4/CoreDisplay/Display/Display.cpp, line 3072.

boot-args:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff2044f946 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff2047e615 pthread_kill + 263
2 libsystem_c.dylib 0x00007fff203d3411 abort + 120
3 libsystem_c.dylib 0x00007fff203d27e8 __assert_rtn + 314
4 com.apple.CoreDisplay 0x00007fff21629026 CoreDisplay::AssertTracer(char const*, char const*, int, unsigned long long, bool, char const*, char const*, ...) + 1378
5 com.apple.CoreDisplay 0x00007fff2166987e CoreDisplay_NotReady + 6154
6 com.apple.SkyLight 0x00007fff24e25f06 update_display_callback(void*, double) + 2414
7 com.apple.SkyLight 0x00007fff24e8f4e2 run_timer_pass + 482
8 com.apple.SkyLight 0x00007fff24e8f610 non_coalesced_timer_handler + 16
9 com.apple.SkyLight 0x00007fff24ecb681 post_port_data + 265
10 com.apple.SkyLight 0x00007fff24ecb2d5 CGXRunOneServicesPass + 1719
11 com.apple.SkyLight 0x00007fff24ecc067 server_loop + 91
12 com.apple.SkyLight 0x00007fff24ecbe39 SLXServer + 1889
13 WindowServer 0x0000000101782320 0x10177f000 + 13088
14 libdyld.dylib 0x00007fff20499f3d start + 1

Thread 1:: com.apple.coreanimation.render-server
0 libsystem_kernel.dylib 0x00007fff2044932a mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff2044969c mach_msg + 60
2 com.apple.QuartzCore 0x00007fff26f6e40c CA::Render::Server::server_thread(void*) + 498
3 com.apple.QuartzCore 0x00007fff26f6e20b thread_fun(void*) + 25
4 libsystem_pthread.dylib 0x00007fff2047e954 _pthread_start + 224
5 libsystem_pthread.dylib 0x00007fff2047a4a7 thread_start + 15

.... snipped (there's more but seems less relevant)

 

[incumbent]

I recently upgraded my old Mac Pro to the 12 core processor, 128GB of RAM, added a new SSD boot drive, installed latest Big Sur now updated to 11.3.1. It's a nice spec but the machine is crashing quite often with all of the usual symptoms - loss of control but audio continues. The crash reports when they appear (for some reason not always) usually have the line:
…
fModelName: 'AMD FirePro D300'
fDeviceTerminated: NO
fGPURestartCount: 14

well fwiw mine has still been running. i've used handbrake in desktop (separate admin account for that) and cli quite a bit, a few other things, but i haven't recreated my user account yet.

haven't seen my crash count get to 14 though usually watchdog bounces me after 4-5 i think.

 

[incumbent]

it bounced again a few hours after my last post above #irony

emory on DreamOn ~
➜ tar tzvf Dropbox/DreamOnFinallyPanic-2021-05-27-logs.tar.gz
-rw-rw----  0 _analyticsd _analyticsusers 766 May 27 19:00 Library/Logs/DiagnosticReports/Analytics-90Day-2021-05-27-190040.8383.core_analytics
-rw-rw----  0 _analyticsd _analyticsusers 139291 May 27 19:00 Library/Logs/DiagnosticReports/Analytics-Daily-2021-05-27-190040.8464.core_analytics
-rw-rw----  0 _analyticsd _analyticsusers    556 May 27 19:00 Library/Logs/DiagnosticReports/Analytics-Journal-90Day-2021-05-27-190040.core_analytics
-rw-rw----  0 _analyticsd _analyticsusers    556 May 27 19:00 Library/Logs/DiagnosticReports/Analytics-Journal-Daily-2021-05-27-190040.835.core_analytics
-rw-rw----  0 _analyticsd _analyticsusers    493 May 27 19:00 Library/Logs/DiagnosticReports/Analytics-Journal-Never-2021-05-27-190040.8372.core_analytics
-rw-rw----  0 _analyticsd _analyticsusers   2540 May 27 19:00 Library/Logs/DiagnosticReports/Analytics-Never-2021-05-27-190040.8785.core_analytics
-rw-rw----  0 root        _analyticsusers  21254 May 27 21:57 Library/Logs/DiagnosticReports/ArqAgent_2021-05-27-215714_DreamOn.cpu_resource.diag
-rw-rw-rw-  0 root        _analyticsusers 5719309 May 27 20:50 Library/Logs/DiagnosticReports/Kernel-2021-05-27-205026.panic
-rw-rw----  0 root        _analyticsusers   40926 May 27 20:43 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204309-1_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40926 May 27 20:43 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204309_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:43 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204319_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:43 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204329_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:44 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204400_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:44 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204410_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:44 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204420_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:44 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204430_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:44 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204450_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:47 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204701_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:47 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204702_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:47 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204711_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:47 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204721_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:47 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204731_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:48 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204801_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:48 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204811_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:48 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204821_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:48 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204831_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:48 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204841_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers   40925 May 27 20:48 Library/Logs/DiagnosticReports/Kernel_2021-05-27-204852_DreamOn.gpuRestart
-rw-rw----  0 root        _analyticsusers 4128694 May 27 20:43 Library/Logs/DiagnosticReports/WindowServer_2021-05-27-204319_DreamOn.userspace_watchdog_timeout.spin
-rw-rw----  0 root        _analyticsusers   68342 May 27 20:43 Library/Logs/DiagnosticReports/WindowServer_2021-05-27-204320_DreamOn.crash
-rw-rw----  0 root        _analyticsusers 4353109 May 27 20:47 Library/Logs/DiagnosticReports/WindowServer_2021-05-27-204757_DreamOn.userspace_watchdog_timeout.spin
-rw-rw----  0 root        _analyticsusers 4342316 May 27 20:48 Library/Logs/DiagnosticReports/WindowServer_2021-05-27-204837_DreamOn.userspace_watchdog_timeout.spin
-rw-rw----  0 root        _analyticsusers     619 May 27 13:15 Library/Logs/DiagnosticReports/awdd_2021-05-27-131500-1_DreamOn.awd
-rw-rw----  0 root        _analyticsusers    5197 May 27 13:15 Library/Logs/DiagnosticReports/awdd_2021-05-27-131500_DreamOn.awd
-rw-rw----  0 root        _analyticsusers   64997 May 27 20:55 Library/Logs/DiagnosticReports/disk writes_2021-05-27-205549_DreamOn.diag
-rw-rw----  0 root        _analyticsusers   66371 May 27 20:51 Library/Logs/DiagnosticReports/mds_stores_2021-05-27-205148_DreamOn.crash
-rw-rw----  0 root        _analyticsusers   66497 May 27 20:51 Library/Logs/DiagnosticReports/mds_stores_2021-05-27-205156_DreamOn.crash
-rw-rw----  0 root        _analyticsusers   49571 May 27 19:37 Library/Logs/DiagnosticReports/send_event_2021-05-27-193748_DreamOn.crash
-rw-rw----  0 root        _analyticsusers   62007 May 27 15:20 Library/Logs/DiagnosticReports/signpost_reporter_2021-05-27-152002_DreamOn.cpu_resource.diag
-rw-rw----  0 root        _analyticsusers   58440 May 27 19:37 Library/Logs/DiagnosticReports/sudo_2021-05-27-193748_DreamOn.crash

Mind you, I was not logged in at-console with my usual user account (emory) but my admin account was logged in and all I could think of doing to keep it busy was run a lot of Handbrake encoding jobs, run a few VirtualBox VMs, that sort of thing. I was going to create a new user account and not restore from backup when I got back to that house next but might not bother anyway.

I'm way out of my lane with this stuff, my professional life is in IT security so most of time is spent doing security research, and I conduct threat modeling workshops, when I have a GPU question I ask the people that run our hash smasher cluster at work! I'm quite comfortable with unix-like environments (I'm extremely familiar with macOS, OpenBSD and FreeBSD but I can get along fine in GNU/Linux, Solaris, AIX and g-d help me IRIX too) but a lot has changed in macOS over the last few releases and I've got a lot of catching up to do,

The first of the gpuRestarts begins like so —

Thu May 27 20:48:52 2021

Event:               GPU Reset
Date/Time:           Thu May 27 20:48:52 2021
Application:
Path:
Tailspin:            /Library/Logs/DiagnosticReports/gpuRestart2021-05-27-204852.tailspin
GPUSubmission Trace ID: 0
OS Version:          Mac OS X Version 11.3.1 (Build 20E241)
Graphics Hardware:   AMD FirePro D300
Signature:           1

Report Data:

GPU Log Version: 2

Restart Channel: 0 GFX

---THE STATE OF THE DRIVER---

AMDRadeonX4000_AMDPitcairnGraphicsAccelerator state: ENABLED
 PCIe Device: [6:0:0], DID=0x6810, RID=0x0, SSID=0x12a
 TotalVideoRAMBytes: 0x0000000080000000 (2147483648)
Uptime 320:27:51.593003

[00] Channel: GFX (HW [00]); last reset at 0:00:00.000000
  CompletedTS = 0x055077da, SubmittedTS = 0x05507801
  Sent to HW: TS = 0x055077db (HW TS = 0x055077dc, WPTR = 0x1ee80) at 320:27:44.350431
  ScheduledTS = 0x055077df, enqueued at 320:27:44.405094
  FirstPendingCB: Process ID = 898, Name = Adobe CEF Helper; SubmitContext = GL (0xffffff93d6c66800)
      GPUAddress = 0x0000000400300000, Size = 0x00000350, VMID = 8
      ContentValidation = SKIP

[01] Channel: Compute0 (HW [01]); last reset at 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000

[02] Channel: Compute1 (HW [02]); last reset at 0:00:00.000000
  CompletedTS = 0x01bf057a, SubmittedTS = 0x01bf057b
  ScheduledTS = 0x01bf057b, enqueued at 320:27:49.394345
   Wait for Channel 3, TS 0x4a48786
   Wait for Channel 4, TS 0x2a65dab
   Wait for Channel 0, TS 0x55077fe
   Wait for Channel 15, TS 0x133dfdc3
[03] Channel: DMA0 (HW [03]); last reset at 0:00:00.000000
  CompletedTS = 0x04a48764, SubmittedTS = 0x04a48789
  ScheduledTS = 0x04a48765, enqueued at 320:27:46.478095
   Wait for Channel 15, TS 0x133dfd64

[04] Channel: DMA1 (HW [04]); last reset at 0:00:00.000000
  CompletedTS = 0x02a65daa, SubmittedTS = 0x02a65dac
  Sent to HW: TS = 0x02a65dab (HW TS = 0x15e45b23, WPTR = 0x48c0) at 320:27:49.394287
  ScheduledTS = 0x02a65dac, enqueued at 320:27:49.394367
   Wait for Channel 2, TS 0x1bf057b
   Wait for Channel 15, TS 0x133dfdc5
  FirstPendingCB: Process ID = 0, Name = kernel_task; SubmitContext = Unknown (0)
      GPUAddress = 0x00000000813bf0a0, Size = 0x00000004, VMID = 0
      ContentValidation = PASS
      Buffer range 0x0 .. 0x3:
      *d0001000  21ca8000  3bff0000  00000000

[05] Channel: UVD (HW [05]); last reset at 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000

[06] Channel: VCE (HW [06]); last reset at 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000

[07] Channel: VCELLQ (HW [07]); last reset at 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000

[08] Channel: SPU (HW [08]); last reset at 0:00:00.000000
  CompletedTS = 0x00000007, SubmittedTS = 0x00000007

[15] Channel: VMPT (HW [04]); last reset at 0:00:00.000000
  CompletedTS = 0x133dfd58, SubmittedTS = 0x133dfdf2
  Sent to HW: TS = 0x133dfd59 (HW TS = 0x15e45b03, WPTR = 0x40c0) at 320:27:44.405305
  ScheduledTS = 0x133dfd79, enqueued at 320:27:46.478817
  FirstPendingCB: Process ID = 0, Name = kernel_task; SubmitContext = Unknown (0)
      GPUAddress = 0x0000000080bc10e0, Size = 0x00000009, VMID = 0
      ContentValidation = PASS
      Buffer range 0x0 .. 0x8:
      *24200040  7c07c100  00000000  000002e7  00000000  2c400000  00000001  00001000
       00000000
Pipe 0: Active, submitted = 7582842, performed = 7582842
 STAT e08 MODE 0 SLAV 0 WSAA 4 WSAAPS 4 DISP 0 PWR 1 MODE 80005000 HA 3440 VA 1440 HS 0 VS 0 PCLK 319750  PD 0 DPCH 0 DPSM 0 LHPT 115122627911

Pipe 1: Not active, submitted = 0, performed = 0
 STAT 600 MODE 4 SLAV 0 WSAA 4 WSAAPS 4 DISP 0 PWR 2  PD 0 DPCH 0 DPSM 0 LHPT 1151226279117011 STAT e7 PWR 0 0 V Counter diff 0 10007 0 SMU 1

Pipe 2: Not active, submitted = 0, performed = 0
 STAT 600 MODE 4 SLAV 0 WSAA 4 WSAAPS 4 DISP 0 PWR 2  PD 0 DPCH 0 DPSM 0 LHPT 1151226279117011 STAT e7 PWR 0 0 V Counter diff 0 10007 0 SMU 1

Pipe 3: Not active, submitted = 0, performed = 0
 STAT 600 MODE 4 SLAV 0 WSAA 4 WSAAPS 4 DISP 0 PWR 2  PD 0 DPCH 0 DPSM 0 LHPT 1151226279117011 STAT e7 PWR 0 0 V Counter diff 0 10007 0 SMU 1

Pipe 4: Not active, submitted = 0, performed = 0
 STAT 600 MODE 4 SLAV 0 WSAA 4 WSAAPS 4 DISP 0 PWR 2  PD 0 DPCH 0 DPSM 0 LHPT 1151226279117011 STAT e7 PWR 0 0 V Counter diff 0 10007 0 SMU 1

Pipe 5: Not active, submitted = 0, performed = 0
 STAT 600 MODE 4 SLAV 0 WSAA 4 WSAAPS 4 DISP 0 PWR 2  PD 0 DPCH 0 DPSM 0 LHPT 1151226279117011 STAT e7 PWR 0 0 V Counter diff 0 10007 0 SMU 1
----------------------------------------
--- DRIVER STATE TRACE---
DeviceID                                : [6:0:0]
----------------------
u64AllAccel                             : 0x0000000000003fff
----------------------
accelStart_entry                        : 1
accelStarted                            : 1
accelStart_exit                         : 1
configureDevice_entry                   : 1
deviceConfigured                        : 1
configureDevice_exit                    : 1
initLinkToPeer                          : 1
createHWHandler                         : 1
createHWInterface                       : 1
accelPowerUpHW_entry                    : 1
accelPowerUpHW                          : 1
accelPowerUpHW_exit                     : 1
bltMgrInitialized                       : 1
atomicBltMgrInitialized                 : 1
----------------------
u64AllHW                                : 0x0000000000ff7fdf
----------------------
hardwareInit_entry                      : 1
hardwareInit                            : 1
hardwareInit_exit                       : 1
initExternalInterfaces                  : 1
hwRegistersInitialized                  : 1
mapDoorbellMemory                       : 0
setupCail                               : 1
initCail                                : 1
setupInitHWCapabilities                 : 1
allocateHWEngines                       : 1
initializeHWEngines                     : 1
powerUpHWEngines                        : 1
startHWEngines                          : 1
initHWInfo                              : 1
hardwarePowerUp_entry                   : 1
hardwarePowerUp                         : 1
hardwarePowerUp_exit                    : 1
initializeTtl                           : 0
hwMemoryInitialized                     : 1
hwGartInitialized                       : 1
hwVmmInitialized                        : 1
initHWDisplay                           : 1
setVirtualSpace                         : 1
initializeHWRegisters                   : 1
----------------------------------------

---THE STATE OF THE GPU---
GPU is connected to PCIe bus and Register access is correct.
 timeSinceLastPowerUp = 320:26:56.800727

[00] HWChannel: GFX, last reset 0:00:00.000000
  CompletedTS = 0x055077db, SubmittedTS = 0x055077df
  PendingTS = 0x55077dc, sent at 320:27:44.350431, AccelChannel: 0, TS = 0x55077db
  Ring: ENABLED
    GPUAddress = 0x0000000080020000, Size = 0x20000, ReadPtr = 0x0001ee00, WritePtr = 0x0001f000

[01] HWChannel: Compute0, last reset 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000
  Ring: ENABLED
    GPUAddress = 0x00000000800c0000, Size = 0x8000, ReadPtr = 0x00000000, WritePtr = 0x00000000

[02] HWChannel: Compute1, last reset 0:00:00.000000
  CompletedTS = 0x01bf057a, SubmittedTS = 0x01bf057a
  Ring: ENABLED
    GPUAddress = 0x0000000080100000, Size = 0x8000, ReadPtr = 0x00003d00, WritePtr = 0x00003d00

[03] HWChannel: DMA0, last reset 0:00:00.000000
  CompletedTS = 0x04a48764, SubmittedTS = 0x04a48764
  Ring: ENABLED
    GPUAddress = 0x0000000080140000, Size = 0x8000, ReadPtr = 0x00005900, WritePtr = 0x00005900
    Command Ring Snapshot:
    0x5880 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x5888 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x5890 : f0000000  f0000000  f0000000  f0000000  f0000000  40200000  209ea180  00010004
    0x5898 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
0x58a0 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x58a8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x58b0 : 60000000  800000c0  00000000  04a48763  30000002  800000d0  800000c0  00000000
    0x58b8 : 00000000  70000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    --- Frame Boundary ---
    0x58c0 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x58c8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x58d0 : f0000000  f0000000  f0000000  f0000000  f0000000  40200000  209ea1c0  00010004
    0x58d8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x58e0 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x58e8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x58f0 : 60000000  800000c0  00000000  04a48764  30000002  800000d0  800000c0  00000000
    0x58f8 : 00000000  70000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    --- Frame Boundary ---
    0x5900 :>f0000000< f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x5908 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x5910 : f0000000  f0000000  f0000000  f0000000  f0000000  40200000  209e4c00  00010004
    0x5918 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x5920 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x5928 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x5930 : 60000000  800000c0  00000000  04a48565  30000002  800000d0  800000c0  00000000
    0x5938 : 00000000  70000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000

[04] HWChannel: DMA1, last reset 0:00:00.000000
  CompletedTS = 0x15e45b02, SubmittedTS = 0x15e45b23
  PendingTS = 0x15e45b03, sent at 320:27:44.405305, AccelChannel: 15, TS = 0x133dfd59
  Ring: ENABLED
    GPUAddress = 0x0000000080160000, Size = 0x8000, ReadPtr = 0x000048c0, WritePtr = 0x000048c0
    Command Ring Snapshot:
    0x4840 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x4848 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x4850 : f0000000  f0000000  f0000000  f0000000  f0000000  40000000  80bc1c82  00009000
    0x4858 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x4860 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x4868 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x4870 : 60000000  80000100  00000000  15e45b22  30000002  80000110  80000100  00000000
    0x4878 : 00000000  70000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
 --- Frame Boundary ---
    0x4880 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x4888 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x4890 : f0000000  f0000000  f0000000  f0000000  f0000000  40000000  813bf0a2  00004000
    0x4898 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48a0 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48a8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48b0 : 60000000  80000100  00000000  15e45b23  30000002  80000110  80000100  00000000
    0x48b8 : 00000000  70000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    --- Frame Boundary ---
    0x48c0 :>f0000000< f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48c8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48d0 : f0000000  f0000000  f0000000  f0000000  f0000000  40000000  80b614a2  00009000
    0x48d8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48e0 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48e8 : f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
    0x48f0 : 60000000  80000100  00000000  15e45924  30000002  80000110  80000100  00000000
    0x48f8 : 00000000  70000000  f0000000  f0000000  f0000000  f0000000  f0000000  f0000000
 Sync: DISABLED TimeSinceSyncDisabled = 1153586800 msec

[05] HWChannel: UVD, last reset 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000
  Ring: ENABLED
    GPUAddress = 0x0000000080180000, Size = 0x0800, ReadPtr = 0x00000000, WritePtr = 0x00000000

[06] HWChannel: VCE, last reset 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000
  Ring: DISABLED
    GPUAddress = 0x00000000801a0000, Size = 0x0800, ReadPtr = 0x00000000, WritePtr = 0x00000000

[07] HWChannel: VCELLQ, last reset 0:00:00.000000
  CompletedTS = 0x00000000, SubmittedTS = 0x00000000
  Ring: DISABLED
    GPUAddress = 0x00000000801c0000, Size = 0x0800, ReadPtr = 0x00000000, WritePtr = 0x00000000

[08] HWChannel: SPU, last reset 0:00:00.000000
  CompletedTS = 0x00000007, SubmittedTS = 0x00000007
  Ring: ENABLED
    GPUAddress = 0x00000000801e0000, Size = 0x0180, ReadPtr = 0x00000024, WritePtr = 0x00000024

Graphic and Compute(GC):
  Graphic Ring: GPUAddress = 0x0000000080020000, Size = 0x00020000, ReadPtr = 0x0001efd0, WritePtr = 0x0001f000
  Compute Ring 1: GPUAddress = 0x00000000800c0000, Size = 0x00008000, ReadPtr = 0x00000000, WritePtr = 0x00000000
    IB: GPUAddress = 0x0000000400060000, ConsumedSize = 0x00000000, RemainSize=0x00000000
  Compute Ring 2: GPUAddress = 0x0000000080100000, Size = 0x00008000, ReadPtr = 0x00003d00, WritePtr = 0x00003d00
    IB: GPUAddress = 0x0000000400220000, ConsumedSize = 0x00000000, RemainSize=0x00000000
  GC Pipeline Status:
    Command Processor(CP): BUSY
      PreFetchParser(PFP): ACTIVE
        PFP Last Processed Commands:
          0xffffffff
          0xc0431000
          0xc0044700
          0xc0001700
          0xc0051000
      MicroEngine(ME): ACTIVE
        ME Last Processed Commands:
          0xffffffff
          0xc0044700
          0xc0001700
          0xc0034300
          0xc0004600
    Graphic Engine Setup: IDLE
    Unified Shader Processor and Input: IDLE
    Texture Pipe: IDLE
    Render Backend: IDLE

DRMDMA: ACTIVE and IDLE
  Ring: GPUAddress = 0x00801400, Size = 0x00008000, ReadPtr = 0x00005900, WritePtr = 0x00005900
    IB: [ENABLE] GPUAddress = 0x00000004209ea1c0, ConsumedSize = 0x00000010

DRMDMA1: ACTIVE and IDLE
  Ring: GPUAddress = 0x00801600 Size = 0x00008000, ReadPtr = 0x000048c0 WritePtr = 0x000048c0
    IB: [ENABLE] GPUAddress = 0x0000000080bc1080, ConsumedSize = 0x00000009

UVD0: Enabled

VCE: Disabled

SAMU: Enabled
  System Domain: NO
    GPUAddress = 0x0000000000000000, VMID = 0
    Failing Protection =
  Virtual Domain: NO
    GPUAddress = 0x0000000000000000, VMID = 0
    Failing Protection =
------------------------
: ** GPU Debug Info Start **
: 0x00006810 : 0x0000011e : 0x00000001 : 0x00000018 : 0x0000a060 : 0x00000001 : 0x00000001 : 0x00006810
: 0x00000001 : 0x00000015 : 0x00000000 : 0x00000001 : 0x00000047 : 0x00005009 : 0x00000001 : 0x00000058
: 0x00000012 : 0x00000001 : 0x00000055 : 0x00000000 : 0x00000001 : 0x0000004a : 0x0000106b : 0x00000001
: 0x0000004d : 0x0000012a : 0x00000000 : 0x00000010 : 0x00010900 : 0x00000000 : 0x00000021 : 0x00000080
: 0x00000000 : 0x00000011 : 0x00000007 : 0x00000002 : 0x0000263d : 0x00000001 : 0x00000002 : 0x000021a0
: 0x84038243 : 0x00000002 : 0x0000219f : 0x00420182 : 0x00000002 : 0x0000219d : 0x00000000 : 0x00000002
: 0x0000219e : 0x00010000 : 0x00000002 : 0x0000219c : 0x00000400 : 0x00000002 : 0x0000a0da : 0x00000000

/* Cut … blah blah */

: ** GPU Debug Info End **
Hardware could not be reset. Further submissions will not be accepted.

Then eventually after a few of those it bounces the whole machine because the GPU failures continued. I can supply all these diagnostics to someone if they want them/need them. As far as DreamOn my little trashcan goes, if WWDC announcements include an Apple Silicon Pro desktop, as long as I can get at least 64GB of RAM in it, I'll buy at least one and turn the trashcan into a household automation server where I am now because my partner definitely will not appreciate me wheeling in a 19" rack like I've got at my other place! Hard to find a lot of fault with the outward appearance macpro6,1 even though we have no problem pointing out it's shortcomings on the inside. If my Homebridge docker container, or kubernetes cluster reboots occasionally it's not a big deal to me. It's when I'm in Lightroom or vscode or Obsidian/DEVONthink actively working that I wanna rage-quit this thing.

It continues. That workstation is still on macOS 11.3, I'm doing a `softwareupdate` now. The biggest kick in the ass about how unreliable this machine has been is that I can't enable FileVault. I float between two cities and when I'm out of town and my photo/audio workstation dies, it sits there waiting for someone to unlock the boot volume and that sucks for a computer that requires unexpected rebooting. It's totally screwed me once already because I couldn't export full-size photos for a lab order, it's a good thing I don't earn a living with my photography.

I'm also experimenting with a virtual remote workstation in Azure that I can hitch up to via RDP and then mount my household storage filers over the Internet (I'll probably use a zerotier software-defined VPN to create a virtual vlan the NAS and cloud desktop are on) and price that out for on-demand use? Probably better off even buying the macpro7,1 except it's appealing that Azure won't vanish for up to two weeks until I get home while traveling.

 

[incumbent]

FYI I had trouble with VMware causing these crashes, but always suspected the issue was some combination of software + hardware and not software alone.

fwiw I didn't have VMWare installed but I installed it because of this message and installed a few VMs (btw Haiku is still being actively developed if any of you liked playing with BeOS!)

didn't seem to have any involvement with my gpu crashes though, but i can't hitch up instrumentation until next week.

 

[incumbent]

I'm going to wait until there are OpenZFS builds that work for Monterey, but I'm totally willing to install a developer preview. It isn't like this machine can get less reliable than it already is. I'm still annoyed there's no m-Series Pro desktop announced. Part of me wants to buy an iMac Pro just to be difficult.

 

[mihai_me]

been having the same issues as described here with my mac pro d700 on Big Sur (also formerly on Catalina).
Crash frequency increased lately to 2-3 times per day, before was 1 time each 3 days or so.
I noticed that either they appeared either after upgrading the OS to Catalina (then to Big Sur) or when I trashed my former monitor and replaced it with a Samsung Odyssey G9 and connected 2 x DP to it + PBP as otherwise there was not enough bandwidth to make the setup work.
I'm not keen on buying 2 x cards from ebay just for an experiment to see if it is hardware or software but did anyone try a hardware replacement so far?
Another side note..on the apple forums a guy told me to reseat the 2 x graphic cards as the contacts there get a bit shabby with time and that most of the time that should fix it. I did not try yet as I'm always running around and this needs that I put 1-2h into it and focus. Will do it soon.