MP 7,1 Mac Pro 2019 NVME RAID drives failing to mount

[mattspace]

Most of the time they both show up, but a couple of times they've both gone AWOL - the first time when coming back out of sleep mode, I think (so I've disabled that) but the second time after a shutdown and reboot without having disconnected the power in between.

I cannot comprehend running something like a Mac Pro without sleep mode enabled - I regularly get 30+ day uptimes, sleeping the machine every night, and often during the day.

Though, I did have to disable the machine's ability to auto-wake to do maintenance etc, so it would stay asleep.

I wonder if there's any correlation between the partial wake that involves, and the PCI bus becoming unsettled.

 

[Radium]

Sorry about that I think I have a solution I've been experienced this with my 2019 Mac Pro for the last 2 years thought I was the only one. Well after buying a Mac Studio for the second music studio no pun intended. We realize how powerful the M2 really is thrashing our Intel Mac to the ground so long and behold we buy a new 2023 Mac Pro to put in 32 TB OWC storage cards. The story gets interesting because we retired a 2009 5.1 Mac Pro that was basically doing stems and mixdowns. While gutting the machine we realized we had an old OWC Excelsior card with 8 TB of storage so for s**** and giggles we installed it in the 2023 PCI slot 8-speed it's been in the machine for about 2 weeks with the 2023 Mac Pro running the latest addition of Sonoma. While the OWC m.28 cards that experience once awhile dropouts and not mounting on the desktop the little Excelsior never had that problem. We've come to the conclusion perhaps the way it was formatted in the 2009 5.1 Mac has something to do with it apfs file system old first generation Samsung m.2 drives 1 terabyte each makes you wonder has something changed with apples raid system software that could be causing the problem because for the few weeks the machine has never dropped the drive regardless where I put it in the machine whatever slot. We're going to do some more testing we've been on the phone with Apple and of course o w c and even sonnet still no Solutions as I stated we were having this problem back when we first got the 2019 but it was very rare and intermediate compared to what's going on now with the latest operating system oh what fun I'll let you know

 

[mattspace]

Sorry about that I think I have a solution I've been experienced this with my 2019 Mac Pro for the last 2 years thought I was the only one. Well after buying a Mac Studio for the second music studio no pun intended. We realize how powerful the M2 really is thrashing our Intel Mac to the ground so long and behold we buy a new 2023 Mac Pro to put in 32 TB OWC storage cards. The story gets interesting because we retired a 2009 5.1 Mac Pro that was basically doing stems and mixdowns. While gutting the machine we realized we had an old OWC Excelsior card with 8 TB of storage so for s**** and giggles we installed it in the 2023 PCI slot 8-speed it's been in the machine for about 2 weeks with the 2023 Mac Pro running the latest addition of Sonoma. While the OWC m.28 cards that experience once awhile dropouts and not mounting on the desktop the little Excelsior never had that problem. We've come to the conclusion perhaps the way it was formatted in the 2009 5.1 Mac has something to do with it apfs file system old first generation Samsung m.2 drives 1 terabyte each makes you wonder has something changed with apples raid system software that could be causing the problem because for the few weeks the machine has never dropped the drive regardless where I put it in the machine whatever slot. We're going to do some more testing we've been on the phone with Apple and of course o w c and even sonnet still no Solutions as I stated we were having this problem back when we first got the 2019 but it was very rare and intermediate compared to what's going on now with the latest operating system oh what fun I'll let you know

Are you suggesting a consideration might be whether the problematic drives were migrated from an older OS and machine, Vs. installed afresh in the new(er) problematic machine?

That would certainly be a datapoint - how many folks experiencing this are experiencing it with drives they had formatted on their current problematic OS version.

My personal suspicion is that no iOS device handles PCI NVME storage, and this is an artefact of macOS internals being replaced with iOS internals over time. Something equivalent to Apple Books on Mac being replaced with a Catalyst port of the iPad version, so now all the proofing infrastructure and publishing workflow is gone on macOS.

 

[H2SO4]

Are you suggesting a consideration might be whether the problematic drives were migrated from an older OS and machine, Vs. installed afresh in the new(er) problematic machine?

That would certainly be a datapoint - how many folks experiencing this are experiencing it with drives they had formatted on their current problematic OS version.

My personal suspicion is that no iOS device handles PCI NVME storage, and this is an artefact of macOS internals being replaced with iOS internals over time. Something equivalent to Apple Books on Mac being replaced with a Catalyst port of the iPad version, so now all the proofing infrastructure and publishing workflow is gone on macOS.

This is my usual MO when upgrading to a new MacPOS:
⌘⌥R boot the destination Mac from its recovery and erase the drive format as APFS and install the latest MacOS.
Use migration assistant to bring across my data.
I have never run a RAID.

I have also had the disappearing drives.

 

[aventham]

I cannot comprehend running something like a Mac Pro without sleep mode enabled - I regularly get 30+ day uptimes, sleeping the machine every night, and often during the day.

Though, I did have to disable the machine's ability to auto-wake to do maintenance etc, so it would stay asleep.

I wonder if there's any correlation between the partial wake that involves, and the PCI bus becoming unsettled.

I should clarify that the only sleep setting I've changed is to turn on the energy saver option to "prevent automatic sleeping when the display is off" - just to eliminate it while attempting my own diagnosis of this problem.
Progress update - I've tried manual bandwidth configuration, and whichever pool I put the cards in, the drives still go AWOL.
So I'm going to try disconnecting the power between boots, as that's the only thing that hasn't failed me yet...

 

[Radium]

Never mind didn't work after a few days it is exhibiting the same problems back to the drawing board

 

[Radium]

Forgot to mention one other thing I noticed when this is going to occur when the Mac is first booting up whether it's the 19 or the 2023 it takes a little longer to get to the login screen that lets me know whether I'm going to have an issue or not. The 2019 Mac Pro under Sonoma was booting up on the first few weeks at an incredible speed and it's very Snappy very happy with it actually. When this drive debacle started I noticed the machine was taking longer to get into the login screen. The 2023 is so fast by the time you hit the power button and sit down the screen is already up, however when the drive is going to be lost it is a few seconds slower don't know if this is related. Take care

 

[aventham]

Forgot to mention one other thing I noticed when this is going to occur when the Mac is first booting up whether it's the 19 or the 2023 it takes a little longer to get to the login screen that lets me know whether I'm going to have an issue or not. The 2019 Mac Pro under Sonoma was booting up on the first few weeks at an incredible speed and it's very Snappy very happy with it actually. When this drive debacle started I noticed the machine was taking longer to get into the login screen. The 2023 is so fast by the time you hit the power button and sit down the screen is already up, however when the drive is going to be lost it is a few seconds slower don't know if this is related. Take care

It's the same with my 2019 - there is always a definite additional delay during the system startup, whenever the drives end up unavailable. The cards themselves always show up, so the system must be recognising those, and then presumably trying but eventually failing to detect/recognise the drives.

 

[H2SO4]

For anybody that's in the Beta program was there been any mention whatsoever about NVME, PCIe or external drives or RAID?

 

[rhoulian]

Here are the 14.4 beta version fixes. No mention of any PCIe problem...

Resolved Issues​

• Fixed an issue where center or right aligned NSTextField appears blurry. (120819010)
• Fixed: Resolves an issue where pointer style does not update when browsing in Safari. (121131986)

Core ML​

New Features​

• ML Program models that are loaded with MLComputeUnits.cpuOnly will use a new high performance CPU backend that takes advantage of Accelerate framework’s BNNS library. (114037934)

CreateML​

Resolved Issues​

• Fixed: When using the transfer learning algorithm option, the CreateML app and framework object detection template might fail to converge and cause poor model quality and produce more than expected false positives. (114480994)

Finder​

Resolved Issues​

• Fixed: Resolves an issue where tiling a window causes the desktop picture to turn black. (118044617)

Messages​

Resolved Issues​

• Fixed: Stickers (Memoji and 3rd party) might appear blank. (120994483)

Passkeys​

Resolved Issues​

• Fixed: Registering passkeys might not work on certain websites. (122217903)

Software Updates​

Resolved Issues​

• Fixed: Updates to macOS 14.4 starting from macOS 11.0–12.3.1 will not work. (120548971)

StoreKit​

New Features​

• productDescriptionHidden(_ API can be used to configure the visibility of product descriptions in ProductView, StoreView and SubscriptionStoreView instances within a view hierarchy. When building with Xcode 15.3, the view modifier can be used even if your app is running on iOS 17.0, iPadOS 17.0, macOS 14.0, tvOS 17.0, watchOS 10.0, visionOS 1.0, or later.
  When implementing a product view style, it can support this new view modifier by checking the descriptionVisibility property on the configuration value. (110414819) (FB12261973)
• You can use SubscriptionStoreView to present promotional offers by adding the subscriptionPromotionalOffer(offer:signature modifier.
  If you’re already using inAppPurchaseOptions(_ modifier to support promotional offers for StoreKit views, you should adopt the new API instead when your app is running on iOS 17.4, iPadOS 17.4, macOS 14.4, tvOS 17.4, watchOS 10.4, visionOS 1.1 or later. Do not use both APIs to apply a promotional offer for the same view. (115358806)

Resolved Issues​

• Fixed: The isEligibleForIntroOffer property and isEligibleForIntroOffer(for method now reflect ineligibility in cases where a customer would otherwise be eligible for the offer if they weren’t actively subscribed. This means a customer which is not currently eligible for an introductory offer may become eligible in the future.
  Customers who redeem an introductory offer for a given subscription group will continue to never be eligible for another introductory offer in that subscription group. You can detect this case this by checking if any one transaction with a matching subscriptionGroupID has the type property on offer set to introductory. (103604770) (FB11889732)

SwiftUI​

New Features​

• Table now supports dynamic numbers of columns with the new TableColumnForEach. (79492167) (FB9189673)

Resolved Issues​

• Fixed: DatePicker in macOS reset focus to the first date component when its bound date changes and timeZone is overriden in the environment. (97376561)

Synchronization​

New Features​

• New os_sync_wait_on_address APIs have been added. They are expected to be used for implementing synchronization primitives that do not have a sense of ownership. Please file feedback if you need Synchronization functionality not covered by existing API. (94759935) (FB10141068)

 

[H2SO4]

Here are the 14.4 beta version fixes. No mention of any PCIe problem...

Resolved Issues​

• Fixed an issue where center or right aligned NSTextField appears blurry. (120819010)
• Fixed: Resolves an issue where pointer style does not update when browsing in Safari. (121131986)

Core ML​

New Features​

• ML Program models that are loaded with MLComputeUnits.cpuOnly will use a new high performance CPU backend that takes advantage of Accelerate framework’s BNNS library. (114037934)

CreateML​

Resolved Issues​

• Fixed: When using the transfer learning algorithm option, the CreateML app and framework object detection template might fail to converge and cause poor model quality and produce more than expected false positives. (114480994)

Finder​

Resolved Issues​

• Fixed: Resolves an issue where tiling a window causes the desktop picture to turn black. (118044617)

Messages​

Resolved Issues​

• Fixed: Stickers (Memoji and 3rd party) might appear blank. (120994483)

Passkeys​

Resolved Issues​

• Fixed: Registering passkeys might not work on certain websites. (122217903)

Software Updates​

Resolved Issues​

• Fixed: Updates to macOS 14.4 starting from macOS 11.0–12.3.1 will not work. (120548971)

StoreKit​

New Features​

• productDescriptionHidden(_ API can be used to configure the visibility of product descriptions in ProductView, StoreView and SubscriptionStoreView instances within a view hierarchy. When building with Xcode 15.3, the view modifier can be used even if your app is running on iOS 17.0, iPadOS 17.0, macOS 14.0, tvOS 17.0, watchOS 10.0, visionOS 1.0, or later.
  When implementing a product view style, it can support this new view modifier by checking the descriptionVisibility property on the configuration value. (110414819) (FB12261973)
• You can use SubscriptionStoreView to present promotional offers by adding the subscriptionPromotionalOffer(offer:signature modifier.
  If you’re already using inAppPurchaseOptions(_ modifier to support promotional offers for StoreKit views, you should adopt the new API instead when your app is running on iOS 17.4, iPadOS 17.4, macOS 14.4, tvOS 17.4, watchOS 10.4, visionOS 1.1 or later. Do not use both APIs to apply a promotional offer for the same view. (115358806)

Resolved Issues​

• Fixed: The isEligibleForIntroOffer property and isEligibleForIntroOffer(for method now reflect ineligibility in cases where a customer would otherwise be eligible for the offer if they weren’t actively subscribed. This means a customer which is not currently eligible for an introductory offer may become eligible in the future.
  Customers who redeem an introductory offer for a given subscription group will continue to never be eligible for another introductory offer in that subscription group. You can detect this case this by checking if any one transaction with a matching subscriptionGroupID has the type property on offer set to introductory. (103604770) (FB11889732)

SwiftUI​

New Features​

• Table now supports dynamic numbers of columns with the new TableColumnForEach. (79492167) (FB9189673)

Resolved Issues​

• Fixed: DatePicker in macOS reset focus to the first date component when its bound date changes and timeZone is overriden in the environment. (97376561)

Synchronization​

New Features​

• New os_sync_wait_on_address APIs have been added. They are expected to be used for implementing synchronization primitives that do not have a sense of ownership. Please file feedback if you need Synchronization functionality not covered by existing API. (94759935) (FB10141068)

A little bit disconcerting but thankyou.

 

[bcomer]

Here are the 14.4 beta version fixes. No mention of any PCIe problem...

Resolved Issues​

• Fixed an issue where center or right aligned NSTextField appears blurry. (120819010)
• Fixed: Resolves an issue where pointer style does not update when browsing in Safari. (121131986)

Core ML​

New Features​

• ML Program models that are loaded with MLComputeUnits.cpuOnly will use a new high performance CPU backend that takes advantage of Accelerate framework’s BNNS library. (114037934)

CreateML​

Resolved Issues​

• Fixed: When using the transfer learning algorithm option, the CreateML app and framework object detection template might fail to converge and cause poor model quality and produce more than expected false positives. (114480994)

Finder​

Resolved Issues​

• Fixed: Resolves an issue where tiling a window causes the desktop picture to turn black. (118044617)

Messages​

Resolved Issues​

• Fixed: Stickers (Memoji and 3rd party) might appear blank. (120994483)

Passkeys​

Resolved Issues​

• Fixed: Registering passkeys might not work on certain websites. (122217903)

Software Updates​

Resolved Issues​

• Fixed: Updates to macOS 14.4 starting from macOS 11.0–12.3.1 will not work. (120548971)

StoreKit​

New Features​

• productDescriptionHidden(_ API can be used to configure the visibility of product descriptions in ProductView, StoreView and SubscriptionStoreView instances within a view hierarchy. When building with Xcode 15.3, the view modifier can be used even if your app is running on iOS 17.0, iPadOS 17.0, macOS 14.0, tvOS 17.0, watchOS 10.0, visionOS 1.0, or later.
  When implementing a product view style, it can support this new view modifier by checking the descriptionVisibility property on the configuration value. (110414819) (FB12261973)
• You can use SubscriptionStoreView to present promotional offers by adding the subscriptionPromotionalOffer(offer:signature modifier.
  If you’re already using inAppPurchaseOptions(_ modifier to support promotional offers for StoreKit views, you should adopt the new API instead when your app is running on iOS 17.4, iPadOS 17.4, macOS 14.4, tvOS 17.4, watchOS 10.4, visionOS 1.1 or later. Do not use both APIs to apply a promotional offer for the same view. (115358806)

Resolved Issues​

• Fixed: The isEligibleForIntroOffer property and isEligibleForIntroOffer(for method now reflect ineligibility in cases where a customer would otherwise be eligible for the offer if they weren’t actively subscribed. This means a customer which is not currently eligible for an introductory offer may become eligible in the future.
  Customers who redeem an introductory offer for a given subscription group will continue to never be eligible for another introductory offer in that subscription group. You can detect this case this by checking if any one transaction with a matching subscriptionGroupID has the type property on offer set to introductory. (103604770) (FB11889732)

SwiftUI​

New Features​

• Table now supports dynamic numbers of columns with the new TableColumnForEach. (79492167) (FB9189673)

Resolved Issues​

• Fixed: DatePicker in macOS reset focus to the first date component when its bound date changes and timeZone is overriden in the environment. (97376561)

Synchronization​

New Features​

• New os_sync_wait_on_address APIs have been added. They are expected to be used for implementing synchronization primitives that do not have a sense of ownership. Please file feedback if you need Synchronization functionality not covered by existing API. (94759935) (FB10141068)

There are almost always extras included that are not included in the notes. This issue is a massive embarrassment for Apple in my eyes.

 

[H. Flower]

Downloading now, fingers crossed the issue is fixed.

If not, well.... at least we have new emojis.

 

[H. Flower]

Upgraded to 14.4, rebooted.

RAID didn't mount.

14.4 doesnt fix the problem.

 

[H2SO4]

Upgraded to 14.4, rebooted.

RAID didn't mount.

14.4 doesnt fix the problem.

Ugh..........

 

[bcomer]

Upgraded to 14.4, rebooted.

RAID didn't mount.

14.4 doesnt fix the problem.

This issue needs its own emoji 💩💩💩💩💩

 

[Radium]

Well if it's any consolation the 2023 Mac Pro exhibits the same problem so at least they won't sweep it under the rug and tell us with our Intel machines to go f ourselves LOL

 

[nopster]

Upgraded to 14.4, rebooted.

RAID didn't mount.

14.4 doesnt fix the problem.

Yesterday late afternoon, I upgraded to 14.4. Rebooted. The drive mounted, but a strange message appeared on top right of the screen: "The disk mounted should be disconnected before shutting down". What????? It's not a usbc/tb/usba disk, it's a card inside with 2 nvme in raid!

Anyway, from 7.30pm to 11pm, the drive remain mounted. Cross fingers. I'll make more tests during this weekend.

 

[H2SO4]

Yesterday late afternoon, I upgraded to 14.4. Rebooted. The drive mounted, but a strange message appeared on top right of the screen: "The disk mounted should be disconnected before shutting down". What????? It's not a usbc/tb/usba disk, it's a card inside with 2 nvme in raid!

Anyway, from 7.30pm to 11pm, the drive remain mounted. Cross fingers. I'll make more tests during this weekend.

That tells me that;
1. The system thinks it's an internal drive.
2. Maybe, the drive disconnected, (as a return of the old problem), and then the system remounted it when it detected it again?

 

[nopster]

That tells me that;
1. The system thinks it's an internal drive.
2. Maybe, the drive disconnected, (as a return of the old problem), and then the system remounted it when it detected it again?

I really didn't understand the message that Sonoma showed. It seems that your 2nd point is what happened. But the Drive icon was there, not disappearing at all. And the message appeared after the reboot of the Sonoma update.

 

[H2SO4]

That tells me that;
1. The system thinks it's an internal drive.
2. Maybe, the drive disconnected, (as a return of the old problem), and then the system remounted it when it detected it again?

I should have added option 3.
It’s a clusterf***

 

[mattspace]

Yesterday late afternoon, I upgraded to 14.4. Rebooted. The drive mounted, but a strange message appeared on top right of the screen: "The disk mounted should be disconnected before shutting down".

I have literally never seen a message that a drive had to be unmounted / disconnected before shutdown in ~30 years of Mac operation.

 

[aventham]

I have literally never seen a message that a drive had to be unmounted / disconnected before shutdown in ~30 years of Mac operation.

Could this be a "fix"? We can have pcie drives but for the sake of stability they have to be mounted after startup and unmounted before shutdown?

 

[H2SO4]

Could this be a "fix"? We can have pcie drives but for the sake of stability they have to be mounted after startup and unmounted before shutdown?

I always do that anyway. Second I log in, I unmount any drives I won't be using and mount/remount as necessary. Not sure that's it.

 

[smckenzie]

I reached out to SonnetTech a few weeks ago re their NVME cards - part of their reply was this:

**Prior to purchase, be aware:
We are tracking an issue with M.2 drives in 2019 and 2023 Mac Pros.
Our engineers have been working hard on this and have now been able to reproduce the issue. Just recently we were able to duplicate this issue with other competitor's hardware as well. From the data we have gathered from other customers, and in our lab, it seems that an update released by Apple in the 2023 time frame for multiple OS releases caused an issue with drives dropping off. We are trying to narrow down exactly which ones.
We have filed a bug report to Apple so that they can get the issue fixed in a future OS release. We are sorry for the inconvenience and are working hard to get it resolved as quickly as possible. We have your case status listed as Waiting for Engineering Fix so that we can alert you once there is a solution.

I asked them to clarify what "dropping off" meant and they said:

By drop-off, we mean that the drive is not seen.
It can give the appearance of a failed drive(s), or, a failed card, or both.

It can be random, or, on restart/power on.

It is also possible the drive may have failed separately, and not been part of this issue.

This issue does not cause damage to the drives, and the drives will be seen normally when used in other computers, or, in an external PCIe to Thunderbolt chassis.

I just had a Crucial SSD in my 7,1 fail after less than 4 months - just couldn't mount it. My OWC SSD which I got a few years ago however has been just fine.