MP 7,1 Mac Pro 2019 NVME RAID drives failing to mount

[mattspace]

Highpoint 1104f arrived, and webhost fell over and is behaving in a way that would fit being locked out of their datacentre. So the nice grey box sits on the table while I try to find a new hosting solution, and rebuild all my mailboxes and sites from backups.

BUT I have a PC, the first windows PC I've owned, which came out of a gamedev studio's decommissioned fleet, so at least I can use it to flash the Samsung SSDs if they need firmware updates.

 

[MisterAndrew]

I haven't read through all of this thread, but is there a solution? My NVMe drives don't always like to appear after a boot. It didn't happen for me before the 14.4 update. I'm on 14.4.1. Here are the adapters I have. The newer one usually shows up. It's only failed to show up once. The older ones rarely show up after the first boot, so I have to restart it once, twice, or more before they pop up.

Older adapter

upHere NVME PCIE 4.0 Adapter,M.2 NVME(M-Key) SSD to PCI Express Adapter with PCI Slot Cover,Support M.2 NVME SSD 2230,2242,2260,2280,Support PCIE 4.0 X4,X8,X16 Slot,with LED Indicator

upHere NVME PCIE 4.0 Adapter,M.2 NVME(M-Key) SSD to PCI Express Adapter with PCI Slot Cover,Support M.2 NVME SSD 2230,2242,2260,2280,Support PCIE 4.0 X4,X8,X16 Slot,with LED Indicator

www.amazon.com

Newer one

GLOTRENDS 22110 M.2 NVMe to PCIe 4.0 X4 Adapter, PCIe 4.0/3.0 X4 Full Speed Support (PA-22110)

GLOTRENDS M.2 NVMe to PCIe 4.0 X4 Adapter for 22110/2280/2260/2242/2230 M.2 NVMe SSD, PCIe 4.0/3.0 X4 Full Speed Support (PA-22110)

www.amazon.com

 

[H2SO4]

A new revision of MacOS is out, does anybody have any details to share?

Apple Seeds Fourth Beta of macOS Sonoma 14.5 to Developers [Update: Public Beta Available]

Apple today seeded the fourth beta of an upcoming macOS Sonoma 14.5 update to developers, with the software coming a week after Apple released the...

www.macrumors.com

 

[mattspace]

FYI The highpoint 1104f and Samsung 990 Pro combo seems stable in Ventura (which a couple of people reported as having problems). SSD temps are about 10c higher than the built in reports, and sleeping power draw has increased by ~9w, but sleep / wake, and reboot cycles haven't shown any issues.

I'm not booting from one of the drives yet, but the two quirks I have in my system are I have access to the power schedule disabled, so my system stays asleep when it's put to sleep, and I have verbose boot permanently enabled.

 

[bcomer]

So 14.5 has been released - any sign of a fix?

 

[Norbert Mikołajczyk]

It seems my issues were solved with 14.4.1. Two weeks with no problems while working on the new project. But without booting into windows 11.

 

[H2SO4]

All I could find was this for the release notes;

macOS Sonoma 14.5 Release Notes | Apple Developer Documentation

Update your apps to use new features, and test your apps against API changes.

developer.apple.com

There must be more than that they addressed surely?

 

[H. Flower]

Most of my issues went away also. the raid usually mounts, I think once or twice it hasn’t. Much better.

 

[bax2023]

Most of my issues went away also. the raid usually mounts, I think once or twice it hasn’t. Much better.

Are you on Catalina 10.15.4 (sig) or latest Sonoma ?

 

[H. Flower]

Latest Sonoma.

 

[profdraper]

All I could find was this for the release notes;

macOS Sonoma 14.5 Release Notes | Apple Developer Documentation

Update your apps to use new features, and test your apps against API changes.

developer.apple.com

There must be more than that they addressed surely?

Well I just installed 14.5 & the Sonnet wouldn’t mount again (after being fine for a while). Then the usual SME/PRAM reset & all good again.

 

[ChrisMakesNMods]

I have been following this thread for a while. I have been having the same issue with my Accelsior 4M2 RAID drive ever since about the time of Sonoma's release – intermittent drive losses with no discernible cause. This was after three years of it consistently working using SoftRAID.

I had a good run for the last few weeks. I thought the issue might be my SSDs overheating. I installed Macs Fan Control and upped the fans to a level just below audible. It seemed to make the problem go away. No issues for several weeks. I was going to post about it and then...

Yesterday I installed 14.5. The system was working after install and several reboots. I shutdown at the end of the day. This morning, after startup, no NVME RAID drives. I couldn't get them back even after an NVRAM reset.

After I shutdown -> pulled the OWC Accelsior card -> restarted -> shutdown again -> remounted the OWC card and one more restart, I did manage to get the drive back momentarily. I moved all the contents (several large sound sample libraries plus Logic) to an external NVME drive and I pulled the OWC card. System seems to be working again, but of course, samples take twice as long to load.

So frustrating. Years of this same setup and after Sonoma the OWC NVME RAID randomly does not mount. I am done trying to fix it. I'll be watching this thread in case there is a miracle. Maybe holy water... hmmm.

My setup:

MacPro 7,1
12-core 3.3 GHz Xeon
96 GB RAM, 12x8GB
1 TB Apple SSD
Radeon Pro 580x
OWC Accelsior 4M2, 4x1TB Samsung 970 EVO Plus (removed)
Sonnet J3i with Windows 10 boot drive (rarely used, but works without problems)

 

[sfalatko]

I have been following this thread for a while. I have been having the same issue with my Accelsior 4M2 RAID drive ever since about the time of Sonoma's release – intermittent drive losses with no discernible cause. This was after three years of it consistently working using SoftRAID.

I had a good run for the last few weeks. I thought the issue might be my SSDs overheating. I installed Macs Fan Control and upped the fans to a level just below audible. It seemed to make the problem go away. No issues for several weeks. I was going to post about it and then...

Yesterday I installed 14.5. The system was working after install and several reboots. I shutdown at the end of the day. This morning, after startup, no NVME RAID drives. I couldn't get them back even after an NVRAM reset.

After I shutdown -> pulled the OWC Accelsior card -> restarted -> shutdown again -> remounted the OWC card and one more restart, I did manage to get the drive back momentarily. I moved all the contents (several large sound sample libraries plus Logic) to an external NVME drive and I pulled the OWC card. System seems to be working again, but of course, samples take twice as long to load.

So frustrating. Years of this same setup and after Sonoma the OWC NVME RAID randomly does not mount. I am done trying to fix it. I'll be watching this thread in case there is a miracle. Maybe holy water... hmmm.

My setup:

MacPro 7,1
12-core 3.3 GHz Xeon
96 GB RAM, 12x8GB
1 TB Apple SSD
Radeon Pro 580x
OWC Accelsior 4M2, 4x1TB Samsung 970 EVO Plus (removed)
Sonnet J3i with Windows 10 boot drive (rarely used, but works without problems)

I too have had similar problems which got a bit better after disabling drive sleep - but they did not go away.

I notice you have 970 EVO Plus drives - I have a mix, a 970 Pro, two 970 EVO Plus, and a 990 Pro. When I had issues with the OWC card the two LEDs for the 970 EVO Plus drives would be flashing.

I have had a bit better luck with a SONNET card but not perfect - again when I loose the NVME it is most consistently the 970 EVO Plus drives.

I have a 990 Pro coming today and I'm going to try and see if removing the 970 EVO Plus drives makes things even better or maybe even solve them (not optimistic as this seems to be more Sonoma than anything).

However, maybe some blades and the 7,1 don't get along. It is like that with the cMP 5,1

Regards,
tfalatko

 

[ChrisMakesNMods]

maybe some blades and the 7,1 don't get along.

I like this thinking. The only reasons I did not replace my NVMEs is because they just worked for 3 years and replacing them would have been an expensive experiment. Let me know if you have any luck after you swap them out.

When I had issues with the OWC card the two LEDs for the 970 EVO Plus drives would be flashing.

My OWC card, Accelsior 4M2, has a massive heatsink that fully covers the NVME drives. Also, the 7,1 has to have the cover on to power up. I think there is a trick to powering it on with the cover off, but I have not learned it yet.

How/where are you seeing these LEDs that are indicating possible faults or errors?

 

[avro707]

Mine are Samsung 980 Pro on the Sonnet card (the silent one with the 4 NVME capability) and I'm not seeing any of these issues since.

One of the 980 Pro NVMEs is the boot drive for Windows 11 Pro for Workstations.

Machine is:

Model Name: Mac Pro
Model Identifier: MacPro7,1
Enclosure: Tower
Processor Name: 28-Core Intel Xeon W
Processor Speed: 2.5 GHz
Number of Processors: 1
Total Number of Cores: 28
L2 Cache (per Core): 1 MB
L3 Cache: 38.5 MB
Hyper-Threading Technology: Enabled
Memory: 48 GB
System Firmware Version: 2022.100.22.0.0 (iBridge: 21.16.5077.0.0,0)
OS Loader Version: 580~2215
Activation Lock Status: Enabled
MacOS 14.5

 

[sfalatko]

I like this thinking. The only reasons I did not replace my NVMEs is because they just worked for 3 years and replacing them would have been an expensive experiment. Let me know if you have any luck after you swap them out.


My OWC card, Accelsior 4M2, has a massive heatsink that fully covers the NVME drives. Also, the 7,1 has to have the cover on to power up. I think there is a trick to powering it on with the cover off, but I have not learned it yet.

How/where are you seeing these LEDs that are indicating possible faults or errors?

The LEDs were visible from the back of the computer - they showed through the bracket. I had 4 blades installed and could see four LEDs - they were normally solid green. When the card disappeared the two 970 EVO Plus LEDs would be blinking and the 970 Pro and the 990 Pro would still be solid though none of the blades would show up in disk utility or the finder.

My second 990 PRO arrived today and I moved the data off the 970 EVO Plus' to the 990 PRO and removed the 970's. I'll monitor and see if the behavior changes with the 970's removed.

Regards,
sfalatko

 

[ChrisMakesNMods]

The LEDs were visible from the back of the computer

Thanks. I never noticed the LEDs at the back of the OWC Accelsior. I am seeing the same behavior. When it is working, all of the LEDs are solid green. If it fails, some or all of the LEDs blink green.

Recovering seems to require a combination of reducing the temperature of the SSDs and getting the system to rescan for devices (NVRAM reset).

Guessing my problem could be heat, SSD sensitivity (as you suggested) or something with how Sonoma manages devices or my 970 EVOs are just baked (3 yrs of cooking).

Mine are Samsung 980 Pro on the Sonnet card (the silent one with the 4 NVME capability) and I'm not seeing any of these issues since.

I have a 4TB Samsung 990 Pro in my ACASIS external drive enclosure. It is also running very hot, 51C, after making it my main drive for my sample libraries. Previously, it was my backup drive for my sample libraries and it ran very cool in the 30s when idle. The only two changes I have made to my system are making it my main sample library drive and upgrading to Sonoma 14.5. Even with no apps open it just cooks. Weird.


I am going to play around with my configuration a bit more. Very strange that the external drive is now running so hot. Maybe one of my app dependancies and related background processes is working it. Nothing I can see, tho. There is very little activity showing in Activity Monitor or iostat for it.

I will update if I find something.

 

[sfalatko]

Thanks. I never noticed the LEDs at the back of the OWC Accelsior. I am seeing the same behavior. When it is working, all of the LEDs are solid green. If it fails, some or all of the LEDs blink green.

Recovering seems to require a combination of reducing the temperature of the SSDs and getting the system to rescan for devices (NVRAM reset).

Guessing my problem could be heat, SSD sensitivity (as you suggested) or something with how Sonoma manages devices or my 970 EVOs are just baked (3 yrs of cooking).


I have a 4TB Samsung 990 Pro in my ACASIS external drive enclosure. It is also running very hot, 51C, after making it my main drive for my sample libraries. Previously, it was my backup drive for my sample libraries and it ran very cool in the 30s when idle. The only two changes I have made to my system are making it my main sample library drive and upgrading to Sonoma 14.5. Even with no apps open it just cooks. Weird.


I am going to play around with my configuration a bit more. Very strange that the external drive is now running so hot. Maybe one of my app dependancies and related background processes is working it. Nothing I can see, tho. There is very little activity showing in Activity Monitor or iostat for it.

I will update if I find something.

I did notice that my 970 EVO Plus drives were running about 4 to 5 degrees C hotter than the 990 Pro and the 970 Pro. But they were generally below 50 C.

 

[mattspace]

I have a 4TB Samsung 990 Pro in my ACASIS external drive enclosure. It is also running very hot, 51C, after making it my main drive for my sample libraries. Previously, it was my backup drive for my sample libraries and it ran very cool in the 30s when idle. The only two changes I have made to my system are making it my main sample library drive and upgrading to Sonoma 14.5. Even with no apps open it just cooks. Weird.

FWIW my dual Evo 990 Pros run at ~35-40c, but my highpoint card (in slot 6) has a fan in the heatsink. System fans are unchanged.

 

[bax2023]

Apple SSDs are Cooler Masters

That one 870 EVO is a bit warmer because I'm currently booting from it for some testing...

Micron 9300 Pro U.2 is 4TB superhot superfast monster.

EDIT1:
BTW Sonoma 14.5 just ignored all NVMEs on Sonnet card, tried two reboots - the same. Rebooted agian to Ventura 13.6.7 - all fine, all drives mounted.

EDIT2: Reboot again to Sonoma 14.5, still no NVMe drives, and again to Ventura, all there. What a ****** OS Sonoma is.

 

[H2SO4]

However, maybe some blades and the 7,1 don't get along. It is like that with the cMP 5,1

Am questioning this assertion because nobody seemed to have the disappearing drive problem before Sonoma?

 

[H2SO4]

Mine are Samsung 980 Pro on the Sonnet card (the silent one with the 4 NVME capability) and I'm not seeing any of these issues since.

One of the 980 Pro NVMEs is the boot drive for Windows 11 Pro for Workstations.

Machine is:

Model Name: Mac Pro
Model Identifier: MacPro7,1
Enclosure: Tower
Processor Name: 28-Core Intel Xeon W
Processor Speed: 2.5 GHz
Number of Processors: 1
Total Number of Cores: 28
L2 Cache (per Core): 1 MB
L3 Cache: 38.5 MB
Hyper-Threading Technology: Enabled
Memory: 48 GB
System Firmware Version: 2022.100.22.0.0 (iBridge: 21.16.5077.0.0,0)
OS Loader Version: 580~2215
Activation Lock Status: Enabled
MacOS 14.5

View attachment 2378676

Interesting. I could copy and paste your first sentences with very minor mods and I do get the problem;
Hardware Overview:

Model Name: Mac Pro
Model Identifier: MacPro7,1
Enclosure: Tower
Processor Name: 12-Core Intel Xeon W
Processor Speed: 3.3 GHz
Number of Processors: 1
Total Number of Cores: 12
L2 Cache (per Core): 1 MB
L3 Cache: 19.3 MB
Hyper-Threading Technology: Enabled
Memory: 64 GB
System Firmware Version: 2022.100.22.0.0 (iBridge: 21.16.4222.0.0,0)
OS Loader Version: 580~1678
Serial Number (system): XXXXXX
Hardware UUID: CE133E07-6D35-5936-98B2-B617C297DF19
Provisioning UDID: CE133E07-6D35-5936-98B2-B617C297DF19
Activation Lock Status: Enabled
MacOS 14.4

980 firmware is 5B2QGXA7.

 

[avro707]

980 firmware is 5B2QGXA7.

Mine are both 2B2QGXA7 - that's a difference.

I'm now on Sonoma 14.5 (the released version). Also no problems. I can't see why it works for me aside from the firmware difference.

 

[sfalatko]

Am questioning this assertion because nobody seemed to have the disappearing drive problem before Sonoma?

I'm not making an assertion - just mentioning that I only seemed to notice problems with my 970 EVO Plus drives not my 970 PRO or 990 PRO drives. As I mentioned some drives and PCIe adapters did not play well with the cMP5,1.

Since I swapped out my 970 EVO Plus drives I can't test that theory. I too have had no problems with 14.5 but remain to be convinced it is fixed.

 

[H2SO4]

Mine are both 2B2QGXA7 - that's a difference.

I’ll check see if there is a revision for mine.

Mine are both 2B2QGXA7 - that's a difference.

I'm now on Sonoma 14.5 (the released version). Also no problems. I can't see why it works for me aside from the firmware difference.

i wish i knew why.
I’ll upgrade to 14.5 after a while, I can’t see how the situation could get any worse.