NVMe with ST-NGFF2013-C; Vega Internal GPU; Mac Pro 2013 (6,1)

[steve123]

Intel says all versions will of PCIe are backward compatible:

What Are PCIe 4.0 and 5.0? - Intel

Newer PCIe standards mean a higher-bandwidth connection to GPUs, SSDs, and other peripherals.

www.intel.ca

and Samsung appears to say so as well:

PCIe 4.0 vs. PCIe 3.0: What's the difference? - Samsung Business Insights

The differences between PCIe 4.0 and PCIe 3.0 are worth knowing. Discover why an upgrade could benefit your devices' performance.

insights.samsung.com

I suspect there is no incompatibility with PCIe. As mentioned earlier, the issue is likely due to some incompatibility between macOS and the Samsung controller not properly handling the power states. Inserting the switch isolates the problem from macOS. It is unfortunate Samsung does not pay closer attention to this problem.

 

[CodeJingle]

I suspect there is no incompatibility with PCIe. As mentioned earlier, the issue is likely due to some incompatibility between macOS and the Samsung controller not properly handling the power states. Inserting the switch isolates the problem from macOS. It is unfortunate Samsung does not pay closer attention to this problem.

Ideal ≠ reality. It's only on paper that PCIe 2 and PCIe 4 are perfectly compatible. In the actual implementation of the standard, there are subtleties.

I wonder if there are similar issues using those drives on any computer with an internal M.2 boot drive over PCIe 2 (including Windows). Part of their guarantee of PCIe 3 compatibility probably comes from testing it. I doubt Samsung is doing any PCIe 2 testing with their drives. We shouldn't expect them to either (to test against all PCIe versions). At least that's my opinion. It's our job as a niche community to figure this stuff out and share our findings.

 

[tsialex]

Intel says all versions will of PCIe are backward compatible:

What Are PCIe 4.0 and 5.0? - Intel

Newer PCIe standards mean a higher-bandwidth connection to GPUs, SSDs, and other peripherals.

www.intel.ca

and Samsung appears to say so as well:

PCIe 4.0 vs. PCIe 3.0: What's the difference? - Samsung Business Insights

The differences between PCIe 4.0 and PCIe 3.0 are worth knowing. Discover why an upgrade could benefit your devices' performance.

insights.samsung.com

I suspect there is no incompatibility with PCIe. As mentioned earlier, the issue is likely due to some incompatibility between macOS and the Samsung controller not properly handling the power states.

If it was just S3, we wouldn't have problems with the blade downgrading to 2,5GT/s when doing a cold boot.

Inserting the switch isolates the problem from macOS. It is unfortunate Samsung does not pay closer attention to this problem.

It's not just so simple. MacPro5,1 slots 3 and 4 share 4 PCIe v2.0 lines by a IDT PCIe v2.0 switch connected to the southbridge lines. A 980/980 PRO blade is almost unusable when installed to slots 3 and 4 and more or less work when installed on slot 2.

The only way to really have it working without any issues is when you install it to a switched adapter connected to PCIe slots 1 or 2.

 

[steve123]

Part of their guarantee of PCIe 3 compatibility probably comes from testing it. I doubt Samsung is doing any PCIe 2 testing with their drives.

If Samsung licensed third party IP for their PCIe serdes then they probably rely on the representations of the third party. Though, I do think that the PCI-SIG has rules about using the trademark that require compliance.

Compliance Program | PCI-SIG

pcisig.com

 

[steve123]

MacPro5,1 slots 3 and 4 share 4 PCIe v2.0 lines by a IDT PCIe v2.0 switch connected to the southbridge lines. A 980/980 PRO blade is almost unusable when installed to slots 3 and 4 and more or less work when installed on slot 2.

The only way to really have it working without any issues is when you install it to a switched adapter connected to PCIe slots 1 or 2.

Interesting ... wonder if we are seeing limitations because the SIG only requires "80%" interoperability?

Did you mention earlier that the 970 EVO Plus works? Sometime in 2021 Samsung changed the controller used in the 970 EVO Plus from the Phoenix controller to the Elpis. The Elpis is the controller used in the 980 Pro. I wonder if newer 970 EVO Plus blades have the same problems as the 980 Pro? Do you know the mfg date of of your 970 EVO plus?

Thinking about the table in the sticky post, I wonder if it would help to include a column with the date so we have an idea of how recent the data is for a particular blade?

 

[CodeJingle]

We are looking for the oldest 4TB M.2 drive to maximize compatibility. But single stick 4TB M.2 drives are ALL pretty much new. They've only been around for a couple of years, right?

I don't have time to do the R&D. If anyone else wants to research which 4TB M.2 SSD was the first to market.

 

[tsialex]

This was completely unexpected, Monterey 12.6.4 beta4 (21G521) have a MacPro6,1 EFI firmware upgrade, from 470.0.0.0 to 474.0.0.0.0:

Hardware Overview:

  Model Name:	Mac Pro
  Model Identifier:	MacPro6,1
  Processor Name:	6-Core Intel Xeon E5
  Processor Speed:	3,5 GHz
  Number of Processors:	1
  Total Number of Cores:	6
  L2 Cache (per Core):	256 KB
  L3 Cache:	12 MB
  Hyper-Threading Technology:	Enabled
  Memory:	16 GB
  System Firmware Version:	474.0.0.0.0
  OS Loader Version:	540.120.3~22
  SMC Version (system):	2.20f18
  Panel Illumination Version:	1.4a6

No news of what this EFI firmware upgrade is about at this moment. There was no developer or any public beta testing for any of the versions between 470.0.0.0.0 and 474.0.0.0.0, unusual, but not unheard when the EFI firmware upgrade was issued for security-related reasons.

Side note, it's one more of the interminable software upgrades, with multiple reboots and taking around 20 to 30 minutes - several minutes with the screen completely off when you think that something went wrong. Just wait and fight the urge of shutting down the Mac Pro.

 

[steve123]

A bit of an update on my 980 PRO experience. I have Big Sur and Monterey installed on separate volumes in the same APFS container. I do not experience extremely long boot times due to TRIM issues when using Big Sur 11.7.4 (boot time < 60 sec) or Monterey 12.5.1 (boot time < 60 sec):

Yesterday I updated Monterey to 12.6.3 and after I did so, I encountered a boot time of more than 5 minutes. Big Sur still boots in less than 60 seconds.

So, it appears something Apple did to Monterey between 12.5.1 and 12.6.3 is causing an issue for me.

With regard to TRIM, the logs show the following:
Monterey 12.6.3:

stevenslupsky@barbie-en0 ~ % log show --last boot --predicate "eventMessage contains 'spaceman'"
Filtering the log data using "composedMessage CONTAINS "spaceman""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       Thread     Type        Activity             PID    TTL
2023-03-23 12:27:29.962736-0600 0x411      Default     0x0                  0      0    kernel: (apfs) spaceman_metazone_init:191: disk1 metazone for device 0 of size 2693771 blocks (encrypted: 0-1346885 unencrypted: 1346885-2693771)
2023-03-23 12:27:29.962742-0600 0x411      Default     0x0                  0      0    kernel: (apfs) spaceman_datazone_init:625: disk1 allocation zone on dev 0 for allocations of 1 blocks starting at paddr 159711232
2023-03-23 12:27:29.962746-0600 0x411      Default     0x0                  0      0    kernel: (apfs) spaceman_datazone_init:625: disk1 allocation zone on dev 0 for allocations of 2 blocks starting at paddr 151683072
2023-03-23 12:27:29.962750-0600 0x411      Default     0x0                  0      0    kernel: (apfs) spaceman_datazone_init:625: disk1 allocation zone on dev 0 for allocations of 3 blocks starting at paddr 165183488
2023-03-23 12:27:29.962754-0600 0x411      Default     0x0                  0      0    kernel: (apfs) spaceman_datazone_init:625: disk1 allocation zone on dev 0 for allocations of 4 blocks starting at paddr 133693440
2023-03-23 12:27:30.069126-0600 0x411      Default     0x0                  0      0    kernel: (apfs) spaceman_scan_free_blocks:3172: disk1 scan took 0.106337 s (no trims)
--------------------------------------------------------------------------------------------------------------------
Log      - Default:          6, Info:                0, Debug:             0, Error:          0, Fault:          0
Activity - Create:           0, Transition:          0, Actions:           0

Big Sur 11.7.4:

barbie-en0:~ stevenslupsky$ log show --last boot --predicate "(process == 'kernel') && (sender == 'apfs') && (eventMessage contains 'trim')"
Filtering the log data using "process == "kernel" AND sender == "apfs" AND composedMessage CONTAINS "trim""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       Thread     Type        Activity             PID    TTL
2023-03-23 13:08:37.213474-0600 0x47b7     Default     0x0                  0      0    kernel: (apfs) spaceman_trim_free_blocks:3371: disk6 scan took 11.219667 s, trims took 7.251637 s
2023-03-23 13:08:37.213480-0600 0x47b7     Default     0x0                  0      0    kernel: (apfs) spaceman_trim_free_blocks:3379: disk6 4055966658 blocks free in 18706 extents
2023-03-23 13:08:37.213487-0600 0x47b7     Default     0x0                  0      0    kernel: (apfs) spaceman_trim_free_blocks:3387: disk6 4055966658 blocks trimmed in 18706 extents (387 us/trim, 2579 trims/s)
2023-03-23 13:08:37.213493-0600 0x47b7     Default     0x0                  0      0    kernel: (apfs) spaceman_trim_free_blocks:3390: disk6 trim distribution 1:14896 2+:2887 4+:689 16+:133 64+:45 256+:56
--------------------------------------------------------------------------------------------------------------------
Log      - Default:          4, Info:                0, Debug:             0, Error:          0, Fault:          0
Activity - Create:           0, Transition:          0, Actions:           0

In the Big Sur case, TRIM was only run when Time Machine mounted its' sparse bundle to create a snapshot. Otherwise, in both cases, there were no trims at boot.

NVMe info shows x4 link width and 5 GT/s speed in both Big Sur and Monterey.

 

[tabormeister]

Has anyone tried OCLP's NVMe fixes? Ventura works great on NVMe on even like 2013 MacBooks with their patches, not sure what they have specifically for nMP (is it still that or now that we have a new new one did it change?) but presumably it's solid

 

[steve123]

Ventura works great on NVMe on even like 2013 MacBooks with their patches, not sure what they have specifically for nMP

I attempted to upgrade the Monterey volume to Ventura 13.2.1 yesterday with OCLP.. The install took a very long time and it appears it crashed a few times during the procedure. Attempted to start it this morning and I am at a black screen.

 

[tabormeister]

I attempted to upgrade the Monterey volume to Ventura 13.2.1 yesterday with OCLP.. The install took a very long time and it appears it crashed a few times during the procedure. Attempted to start it this morning and I am at a black screen.

Weird. I kinda want an nMP to play with now

 

[steve123]

Weird. I kinda want an nMP to play with now

I have made some progress and I have successfully booted into Ventura but performance is very "stop and go". I clearly see why others have reported problems with the 980 PRO. I am beginning to consider that the problem with the 980 PRO may be an incompatibility with APFS disk encryption. I am in the process of decrypting the APFS encrypted volume used for Ventura. The decryption should be complete in a few more hours.

 

[tsialex]

Has anyone tried OCLP's NVMe fixes? Ventura works great on NVMe on even like 2013 MacBooks with their patches, not sure what they have specifically for nMP (is it still that or now that we have a new new one did it change?) but presumably it's solid

2013 MacBooks Pro are Haswell, with required full AVX2.0 support, while late-2013 Mac Pro supports only AVX1.0. There are so many problems with Ventura and late-2013 Mac Pro that the only workaround to run it is via a VM, without any Power Management and DRM support. Take a look at the reference thread:

https://forums.macrumors.com/threads/macpro6-1-ventura-and-oclp-still-no-power-management-and-no-drm.2367706/

 

[tsialex]

I am beginning to consider that the problem with the 980 PRO may be an incompatibility with APFS disk encryption.

You could be to something, FV2 could certainly exacerbate the issues we have, but I personally never used FV2 with my late-2013 Mac Pro and long relegated the 980 and 980 PRO to my MacPro5,1 with HighPoint SSD7101A-1 because of the freezes and beachballs that I don't have with the 970 EVO Plus or 970 Pro.

 

[tabormeister]

You could be to something, FV2 could certainly exacerbate the issues we have, but I personally never used FV2 with my late-2013 Mac Pro and long relegated the 980 and 980 PRO to my MacPro5,1 with HighPoint SSD7101A-1 because of the freezes and beachballs that I don't have with the 970 EVO Plus or 970 Pro.

I bet P31 Gold would work well too, it's been super smooth in ever Mac I've tried and sips power/runs very cool (admittedly all haswell but still, I bet it would work well.) It's like $120 for 2tb right now as well which is CRAZY considering it's by far the best pcie 3 ssd

 

[steve123]

I bet P31 Gold would work well too

Yes, that looks like a good SSD and it has a good rating in the PCIe SSD Wikipost:

PCIe SSDs - NVMe & AHCI

This is a general info thread for blade SSDs that can be used in the Mac Pro. This is a WikiPost so anyone with the proper credentials may edit it. NVMe SSDs can be used as a boot drive in the MP5,1 and MP6,1 with the latest firmware installed (beginning with version 140.0.0.0.0 for MP5,1 and...

forums.macrumors.com

Unfortunately, it is coming up much more expensive for me here (Amazon.ca).

Ventura is booting but boot time is still very long (about 6 minutes). On the latest boot, I did actually see a TRIM in the log ... though I am suspicious about the amount of time reported. I rebooted the computer this morning and when I came back to the computer, it was sleeping. I woke it up at 9:05 and logged in and the boot log shows that at 9:12 the TRIM took 1402 seconds. That doesn't really make sense to me because if the TRIM completed at 9:12, then it must have started about 23 min earlier at about 8:49. But, it was sleeping at 9:05 so I am not sure how to interpret this:

stevenslupsky@barbie-en0 ~ % log show --last boot --predicate "(process=='kernel') && (sender=='apfs') && (eventMessage contains 'trim')"
Filtering the log data using "process == "kernel" AND sender == "apfs" AND composedMessage CONTAINS "trim""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       Thread     Type        Activity             PID    TTL 
2023-03-25 08:06:25.740836-0600 0x2f6      Default     0x0                  0      0    kernel: (apfs) spaceman_scan_free_blocks:3311: disk1 scan took 0.112813 s (no trims)
2023-03-25 09:12:15.092452-0600 0x225      Default     0x0                  0      0    kernel: (apfs) spaceman_scan_free_blocks:3293: disk1 scan took 1403.118732 s, trims took 1401.758457 s
2023-03-25 09:12:15.092471-0600 0x225      Default     0x0                  0      0    kernel: (apfs) spaceman_scan_free_blocks:3303: disk1 96754095 blocks trimmed in 681193 extents (2057 us/trim, 485 trims/s)
2023-03-25 09:12:15.092475-0600 0x225      Default     0x0                  0      0    kernel: (apfs) spaceman_scan_free_blocks:3306: disk1 trim distribution 1:198897 2+:171529 4+:161646 16+:68337 64+:36805 256+:43979

FYI, I have the NVMe Power Management setting turned on in OCLP:

I am authoring this post in Ventura and it seems stable now after several boots and applying the OCLP Post Install Root Patch. Long boot time though and I have only been running for an hour so it is very premature to tell whether or not the system is stable.

I have a spare Kingston Fury Renegade 1TB I was planning to use with an external enclosure. I think I will pull the 980 PRO ... it just doesn't seem to be working out for me very well and the debugging is not making much progress. I was trying to avoid pulling it because I don't have the enclosure yet to help me copy my files over. I will have to resort to using a networked TM backup which takes a long time. I may wait for the coming 13.3 release before pulling the 980 PRO. Maybe the regression that appeared in Monterey will be fixed (anyone tried a 980 PRO with a 13.3 beta?).

 

[steve123]

long relegated the 980 and 980 PRO to my MacPro5,1 with HighPoint SSD7101A-1 because of the freezes and beachballs that I don't have with the 970 EVO Plus or 970 Pro.

I am running a similar machine as you are:

I notice a lot of "stop and go" during boot and after login for several minutes. The graph in the screen shot of the Activity Monitor below shows CPU load at several points where the system is entirely locked up (no mouse or graphical UI updates for up to two or three minutes):

Eventually, it seems to settle down and the "stop and go" and beachballs go away but I've only been tinkering around for about an hour. I also noticed that Activity Monitor shows the kernel task running at 100% continuously even after the "settle down" period. Is that because Ventura is running in a VM?

 

[tabormeister]

I am running a similar machine as you are:
View attachment 2178584
I notice a lot of "stop and go" during boot and after login for several minutes. The graph in the screen shot of the Activity Monitor below shows CPU load at several points where the system is entirely locked up (no mouse or graphical UI updates for up to two or three minutes):
View attachment 2178585

Eventually, it seems to settle down and the "stop and go" and beachballs go away but I've only been tinkering around for about an hour. I also noticed that Activity Monitor shows the kernel task running at 100% continuously even after the "settle down" period. Is that because Ventura is running in a VM?

OCLP is not a VM, but an EFI shim/bootloader with some other kext fixes, workarounds and other tweaks that make booting old Macs with newer MacOS relatively painless compared to other methods. Virtual machine would be running Ventura entirely virtually within another operating system (like Monterey) as the host

 

[steve123]

OCLP is not a VM

My understanding is that on the Mac Pro 6,1, OCLP runs macOS in a VM:

OCLP 0.5.1 Nightly brought Ventura support for the late-2013 Mac Pro. Somethings don't yet work, like DRM and CPU power management (Ventura is running with the VMM flag enabled), but besides that, it's fast/stable and the installation is fully automatic:

I also noticed that since installing Ventura, a mysterious volume named "VM" has appeared when I list the volumes:

    +-> Volume disk1s8
        ---------------------------------------------------
        APFS Volume Disk (Role):   disk1s8 (VM)
        Name:                      VM (Case-insensitive)
        Mount Point:               /System/Volumes/VM
        Capacity Consumed:         1069056 B (1.1 MB)
        Sealed:                    No
        FileVault:                 No

EDIT: VM is "normal" for OCLP.

 

[tabormeister]

My understanding is that on the Mac Pro 6,1, OCLP runs macOS in a VM:

Oh weird, I think you're right. Seems like a hardware level hypervisor thing, wild

 

[steve123]

Here is another view of the CPU load by core:

 

[aldergood]

This was completely unexpected, Monterey 12.6.4 beta4 (21G521) have a MacPro6,1 EFI firmware upgrade, from 470.0.0.0 to 474.0.0.0.0:

Hardware Overview:

  Model Name:    Mac Pro
  Model Identifier:    MacPro6,1
  Processor Name:    6-Core Intel Xeon E5
  Processor Speed:    3,5 GHz
  Number of Processors:    1
  Total Number of Cores:    6
  L2 Cache (per Core):    256 KB
  L3 Cache:    12 MB
  Hyper-Threading Technology:    Enabled
  Memory:    16 GB
  System Firmware Version:    474.0.0.0.0
  OS Loader Version:    540.120.3~22
  SMC Version (system):    2.20f18
  Panel Illumination Version:    1.4a6

No news of what this EFI firmware upgrade is about at this moment. There was no developer or any public beta testing for any of the versions between 470.0.0.0.0 and 474.0.0.0.0, unusual, but not unheard when the EFI firmware upgrade was issued for security-related reasons.

Side note, it's one more of the interminable software upgrades, with multiple reboots and taking around 20 to 30 minutes - several minutes with the screen completely off when you think that something went wrong. Just wait and fight the urge of shutting down the Mac Pro.

Wow. Sound it not really a good upgrade if there're not much things that need to be patched. Since 470.0.0.0, the system seems much more stable now.

 

[aldergood]

I upgrade to 1TB using SKHynix Gold P31 1 TB and everything works perfectly. FW of drive is 41062C20 and is the latest one as of Mar 2023. I forget to run the disk bench mark when using Apple SSD for comparison but have feeling tha apps is running faster.

 

[steve123]

Swapped out the Samsung 980 PRO for a Kingston Fury Renegade. TRIM times are an order of magnitude less time and there is no "stop and go" behaviour.

Installed the 980 PRO into an external USB4 enclosure. Since PCIe is tunnelled over the USB4 interface, I notice that macOS System Info includes the SSD in the NVMe panel and TRIM is enabled. Are there still TRIM issues with the 980 PRO when used in an external USB4 enclosure?

 

[tabormeister]

Swapped out the Samsung 980 PRO for a Kingston Fury Renegade. TRIM times are an order of magnitude less time and there is no "stop and go" behaviour.

Installed the 980 PRO into an external USB4 enclosure. Since PCIe is tunnelled over the USB4 interface, I notice that macOS System Info includes the SSD in the NVMe panel and TRIM is enabled. Are there still TRIM issues with the 980 PRO when used in an external USB4 enclosure?

Good question. What speeds do you get that way? If you have it in a thunderbolt port it likely is running at pretty good speeds