Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
NVMe with ST-NGFF2013-C; Vega Internal GPU; Mac Pro 2013 (6,1)
- Thread starter CodeJingle
- Start date
- Sort by reaction score
Even if I wasn't having a crash issue, it is a bad sign that the latest firmware for Mac Pro (Late 2013) won't install if the internal hard drive is upgraded to current generation NVMe.
[doublepost=1531177213][/doublepost]
when did you swap your internal hard drive for nvme? what is your current firmware revision?
the latest firmware is in macos high sierra 10.13.5, released to market june 1, though was available as early as april 3 through developer beta (97 days ago). it is possible your firmware was upgraded before you swapped.
Last edited:
Firmware upgrades should happen regardless of what you're installing to; you can install the OS to another SSD via USB on the Mac Pro and the firmware update comes free, as long as you're doing all the installing on the Mac Pro you want to update firmware for. Internal or external, HDD or SSD doesn't matter.
I know this because it was a trick I used to get firmware updates before my company upgraded from Sierra to High Sierra.
I upgraded to MP61.0124.B00 from MP61.0120.B00, which is what I got back from the Apple Store in late May. Did the nVMe update first thing, then updated to the latest High Sierra developer beta then Mojave dev beta.
I know this because it was a trick I used to get firmware updates before my company upgraded from Sierra to High Sierra.
I upgraded to MP61.0124.B00 from MP61.0120.B00, which is what I got back from the Apple Store in late May. Did the nVMe update first thing, then updated to the latest High Sierra developer beta then Mojave dev beta.
it is still a problem, to upgrade a machine and still need to keep all the old hardware around for the purposes of updating firmware. if i update the internal graphics card to vega and it works but i have to keep the old graphics card nearby and swap it out or connect it via thunderbolt to update the firmware that is bullocks.
if it is true that any drive can be the internal drive for updating firmware as long as the booting drive is stock ssd, this is different than what i have heard previously. thanks!
but i am still unstable on mojave. anyone else care to chime in with their recent experiences if they are running the latest macos beta and latest firmware with their internal drive being nvme.
The best description of the current situation with EFI graphics support and storage support is certainly less polite than "bullocks".
Does any "current standard" GPU or SSD work out of the box on an Apple?
The industry dropped "EFI" for "UEFI" in 2005 - but Apple continues to use a bastardized version of "EFI". Laziness, or lockout?
If Apple supported UEFI, every GPU would have a boot screen. Wouldn't that be a good thing?
i keep using bullocks because it isn’t censored. for egpu i would say yes. for internal gpu no, support for upgrade is very limited. i’m still mapping the pins for the internal gpu, it is taking so long it’s more of an academic project at this point. coloring in the lines is really boring.
yeah your firmware seems slightly behind.
[doublepost=1531254122][/doublepost]
Kris Kelvin has also been having issues with NVMe and more recent version of macOS (where a previous version worked fine), so I am not the only one. My NVMe partitions may be corrupted because of the kernel panic during partition resizing, so I'll have to do a 100% clean install of mojave, but also now with the latest version of macOS and firmware the machine seems to be crashing when booting from a USB 3 drive.
Last edited:
I'll report back if I find anything weird or malfunctioning on my end. Still not running into problems with a Samsung 960 EVO 500 GB (small correction from before; it's not a power of two storage, it is actually 500 GB even).
This Mac Pro was a pre-owned purchase from a pawn shop on eBay. Mojave dev beta 3 (18A326g) has been the most stable macOS I've seen so far, and the nVMe upgrade didn't happen until I got it back from the Apple Store for repairs.
I did encounter the GPU freeze many times a day up until the latest Mojave, even after repairs. I've also built LLVM + Clang from scratch overnight on three occasions now without issues barring new Xcode 10 beta and the new placement of C++ headers, so the hardware looks good.
[doublepost=1531260156][/doublepost]Just in case any of my variations are making a difference:
1) I do have the heatsink on the 960 EVO, with both thermal pads applied to both ends of the nVMe drive. This did initially cause the drive to stick out of the adapter more than desired, so I ended up compensating with kapton tape (hah). Do not try this at home.
2) I have smcFanControl revving up the minimum main fan speed on the Mac Pro to 1500rpm, as it's usually set incredibly low. Mostly to keep the internals cool and avoid heat damage in using non-stock components.
3) I have been able to boot to external drives via USB, I just tried ASD EFI 3S159 for some unnecessary diagnostic fun in Apple Platinum. No issues found from there.
This Mac Pro was a pre-owned purchase from a pawn shop on eBay. Mojave dev beta 3 (18A326g) has been the most stable macOS I've seen so far, and the nVMe upgrade didn't happen until I got it back from the Apple Store for repairs.
I did encounter the GPU freeze many times a day up until the latest Mojave, even after repairs. I've also built LLVM + Clang from scratch overnight on three occasions now without issues barring new Xcode 10 beta and the new placement of C++ headers, so the hardware looks good.
[doublepost=1531260156][/doublepost]Just in case any of my variations are making a difference:
1) I do have the heatsink on the 960 EVO, with both thermal pads applied to both ends of the nVMe drive. This did initially cause the drive to stick out of the adapter more than desired, so I ended up compensating with kapton tape (hah). Do not try this at home.
2) I have smcFanControl revving up the minimum main fan speed on the Mac Pro to 1500rpm, as it's usually set incredibly low. Mostly to keep the internals cool and avoid heat damage in using non-stock components.
3) I have been able to boot to external drives via USB, I just tried ASD EFI 3S159 for some unnecessary diagnostic fun in Apple Platinum. No issues found from there.
thanks please do. i have heard kernel panics occuring during heavy load when using xcode 10 beta. my mac pro is inside out so the issue is not heat related. i have many heatsinks and fans controlled independently of how much heat is being produced. yes the nvme has a heatsink. kapton tape tends to trap heat so be careful using it next to a heatsink. sure i can try asd to check for hardware failure though i don't get the 'apple platinum' reference, unless you are saying asd is built on top of a base platform known as apple platinum.
Last edited:
I'm passing all ASD EFI 3S159 tests but failing at least one of the ASD OS 3S159 tests. I can test again after swapping out NVMe for stock SSD. Hard drive tests do not exist I am assuming this is because NVMe I hope after swapping out to stock SSD that some hard drive tests show up.
I guess memory tests are really slow with 64 gigabytes.
I guess memory tests are really slow with 64 gigabytes.
Last edited:
yeah for ASD EFI the hard drive tests only show up if a stock drive is detected. my machine passes 88 ASD EFI tests. if there are more than 88 let me know. two of the 88 i can only run when the stock drive is detected.
Has anyone tried a
SSD 970 EVO NVMe M.2 1TB?
I can find them for a bit cheaper than the 960 evo pro.
Cheers.
Colin
SSD 970 EVO NVMe M.2 1TB?
I can find them for a bit cheaper than the 960 evo pro.
Cheers.
Colin
thecowgoddess has, her response is earlier in the thread. try DMing her.
[doublepost=1531422721][/doublepost]It looks like the ASD OS 3S159 tests are meant to be run with the display connected to hdmi. If you are actively using a thunderbolt port it messes with the thunderbolt tests. I'm passing all tests now (so far) running with display hooked up to hdmi and all thunderbolt ports unused.
Last edited:
i continue to kernel panic on nvme io even though the asd hardware tests are passing. this is the consistent source of kernel panic, or at least something that looks similar (definitely always involving IONVMe and AppleMobileFileIntegrity). i need to upgrade my backup drive before i can backup, re-partition, and do a clean install of macos to rule out hard drive corruption. unless the nvme drive has a hardware failure. but the nvme drive is only six months old so hardware failure is a fallback diagnosis for later on. i'm still thinking the kernel panic while resizing the main apfs partition/volume in boot camp assistant damaged the partition/volume in a way that is not repairable except to delete the partition/volume and make a new one.
*** Panic Report ***
panic(cpu 0 caller 0xffffff7f88d97086): nvme: "Fatal error occurred. CSTS=0x3
. FW Revision=2B6QCXP7\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.200.19/IONVMeController.cpp:5278
Backtrace (CPU 0), Frame : Return Address
0xffffffa787ff3a20 : 0xffffff80055a6fdd
0xffffffa787ff3a70 : 0xffffff80056e04a3
0xffffffa787ff3ab0 : 0xffffff80056d1f1a
0xffffffa787ff3b20 : 0xffffff8005554d50
0xffffffa787ff3b40 : 0xffffff80055a69f7
0xffffffa787ff3c60 : 0xffffff80055a6843
0xffffffa787ff3cd0 : 0xffffff7f88d97086
0xffffffa787ff3e30 : 0xffffff8005c578f7
0xffffffa787ff3ea0 : 0xffffff8005c57819
0xffffffa787ff3ed0 : 0xffffff80055e28f4
0xffffffa787ff3f40 : 0xffffff80055e2485
0xffffffa787ff3fa0 : 0xffffff80055540ce
Kernel Extensions in backtrace:
com.apple.iokit.IONVMeFamily(2.1)[D1CC06F9-FFEE-31B4-8E50-B23EFA17EEF8]@0xffffff7f88d82000->0xffffff7f88dc0fff
dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[B9A4167D-7875-393E-AAE2-6DCE8C3964BA]@0xffffff7f86512000
dependency: com.apple.iokit.IOPCIFamily(2.9)[8FFA7106-3C07-3AF6-BFAC-3481EF78689B]@0xffffff7f85e95000
dependency: com.apple.driver.AppleEFINVRAM(2.1)[72552A03-D771-3D6E-B6DE-F3DA267689CE]@0xffffff7f866a7000
dependency: com.apple.iokit.IOStorageFamily(2.1)[75B6AF84-BEA5-3D99-994D-7CA156029ACF]@0xffffff7f8630d000
dependency: com.apple.iokit.IOReportFamily(31)[5C87D64F-B62C-3FBA-B37E-94FE31F1F4F6]@0xffffff7f86b3b000
[doublepost=1531437798][/doublepost]my windows vm is over 300 gigabytes so if i go back to stock ssd i need to buy terabyte. that will be a last resort. first i'll try wiping the drive and do a clean install of mojave beta version 3.2.
*** Panic Report ***
panic(cpu 0 caller 0xffffff7f88d97086): nvme: "Fatal error occurred. CSTS=0x3
. FW Revision=2B6QCXP7\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.200.19/IONVMeController.cpp:5278
Backtrace (CPU 0), Frame : Return Address
0xffffffa787ff3a20 : 0xffffff80055a6fdd
0xffffffa787ff3a70 : 0xffffff80056e04a3
0xffffffa787ff3ab0 : 0xffffff80056d1f1a
0xffffffa787ff3b20 : 0xffffff8005554d50
0xffffffa787ff3b40 : 0xffffff80055a69f7
0xffffffa787ff3c60 : 0xffffff80055a6843
0xffffffa787ff3cd0 : 0xffffff7f88d97086
0xffffffa787ff3e30 : 0xffffff8005c578f7
0xffffffa787ff3ea0 : 0xffffff8005c57819
0xffffffa787ff3ed0 : 0xffffff80055e28f4
0xffffffa787ff3f40 : 0xffffff80055e2485
0xffffffa787ff3fa0 : 0xffffff80055540ce
Kernel Extensions in backtrace:
com.apple.iokit.IONVMeFamily(2.1)[D1CC06F9-FFEE-31B4-8E50-B23EFA17EEF8]@0xffffff7f88d82000->0xffffff7f88dc0fff
dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[B9A4167D-7875-393E-AAE2-6DCE8C3964BA]@0xffffff7f86512000
dependency: com.apple.iokit.IOPCIFamily(2.9)[8FFA7106-3C07-3AF6-BFAC-3481EF78689B]@0xffffff7f85e95000
dependency: com.apple.driver.AppleEFINVRAM(2.1)[72552A03-D771-3D6E-B6DE-F3DA267689CE]@0xffffff7f866a7000
dependency: com.apple.iokit.IOStorageFamily(2.1)[75B6AF84-BEA5-3D99-994D-7CA156029ACF]@0xffffff7f8630d000
dependency: com.apple.iokit.IOReportFamily(31)[5C87D64F-B62C-3FBA-B37E-94FE31F1F4F6]@0xffffff7f86b3b000
[doublepost=1531437798][/doublepost]my windows vm is over 300 gigabytes so if i go back to stock ssd i need to buy terabyte. that will be a last resort. first i'll try wiping the drive and do a clean install of mojave beta version 3.2.
Last edited:
I ordered ANOTHER external enclosure for stock SSD (I gave away my previous enclosure because I didn't think I needed it anymore). This will enable me to more easily attempt to repair my NVMe partition/volume (it may have been damaged after a kernel panic during partition/volume resize while running boot camp assistant). Otherwise, I can try backing up the data and wiping the drive to remove the damaged partition/volume (as long as the damage is software-only and not hardware). I have a four terabyte external drive for backup I can try making room on that otherwise I saw a decent priced eight terabyte external drive if I need to upgrade.
https://eshop.macsales.com/item/OWC/MAU3ENPRPCIO/
https://eshop.macsales.com/item/OWC/MAU3ENPRPCIO/
Last edited:
I am not a big fan of the OWC Envoy Pro. 1st, it's USB and secondly it has issues: Apple OEM SSD manufactured by TOSHIBA are not working at all when mounted in this enclosure.
unfortunately, the one I like is not available without an SSD in it, but it ticks all the boxes -> Transcend JetDrive 825
https://www.transcend-info.com/Products/No-850
unfortunately, the one I like is not available without an SSD in it, but it ticks all the boxes -> Transcend JetDrive 825
https://www.transcend-info.com/Products/No-850
i have had zero issues with the envoy and unfortunately i've already paid for it, so i'll go with envoy for now. the jetdrive is also very expensive, 300+ usd, and the envoy is 70+ usd.
jetdrive 825
envoy
owc does have a brand new thunderbolt 3 single drive external enclosure for m.2 but it fails in the same way (the enclosure is sold only with ssd). i imagine inside the enclosure it is also probably compatible only with regular m.2 connector not proprietary apple m.2 equivalent.
https://eshop.macsales.com/item/OWC/TB3ENVPRC02
continuing in the spirit of alternate solutions, i was interested in other offerings for m.2 external enclosure. this one looks nice for connecting four m.2 drives in raid with external thunderbolt 3 interface. the 'express 4m2'
https://eshop.macsales.com/shop/express-4m2
Last edited:
except the thermal interface test all automated asd efi and asd os tests are passing. at least whatever tests are showing up when i run asd. 88 out of 88 tests on asd efi and 152 out of 153 on asd os. though i have reason to believe some tests are missing from my skew of asd. for example the asd documentation talks of a 'hard drive full scan' test in asd efi but in my skew i am only seeing 'hard drive check smart status' test and 'hard drive short random multi-block' test. about 6 hours and 20 minutes for me to run all asd tests.
i'm worried why my computer is force shutting down during the thermal interface test. cpu_prochot lights up about a minute before the forced shutdown but the system is not overheating. that light coming on when the system is not overheating can only happen when the total system power is exceeded. with the dual le grand macho rt the cooling tdp is ~640 so the cpu will never throttle due to a heat issue. if the thermal interface test is pushing the cpu into turbo boost mode it will stay in turbo boost for the entire test. the ability of the power supply to drive sustained turbo boost and dual D700s may be limited.
i'm using a non-stock cpu, but its power use is only 20 watts more than the highest end stock cpu. bloody hell if apple didn't design 20 watts of overhead in the mac pro power supply. they should have tested in the freezer or antarctica where turbo boost can be sustained.
it looks like the mac pro has lousy thermals and a lousy power supply. there is no way this machine was ever going to support gpu upgrade past dual d700s unless the new gpus emitted the same amount of heat and used the same amount of electricity, which would only be possible if the new gpus have smaller manufacturing process. and not just once - every gpu upgrade requires a smaller manufacturing process than the previous upgrade. examining the timeline of desktop gpus this is not a reasonable assumption, unless apple planned to update the mac pro gpus only once every 5 or 10 years, or if apple originally planned to release their own desktop gpu chips in time for a mac pro upgrade.
i'm worried why my computer is force shutting down during the thermal interface test. cpu_prochot lights up about a minute before the forced shutdown but the system is not overheating. that light coming on when the system is not overheating can only happen when the total system power is exceeded. with the dual le grand macho rt the cooling tdp is ~640 so the cpu will never throttle due to a heat issue. if the thermal interface test is pushing the cpu into turbo boost mode it will stay in turbo boost for the entire test. the ability of the power supply to drive sustained turbo boost and dual D700s may be limited.
i'm using a non-stock cpu, but its power use is only 20 watts more than the highest end stock cpu. bloody hell if apple didn't design 20 watts of overhead in the mac pro power supply. they should have tested in the freezer or antarctica where turbo boost can be sustained.
it looks like the mac pro has lousy thermals and a lousy power supply. there is no way this machine was ever going to support gpu upgrade past dual d700s unless the new gpus emitted the same amount of heat and used the same amount of electricity, which would only be possible if the new gpus have smaller manufacturing process. and not just once - every gpu upgrade requires a smaller manufacturing process than the previous upgrade. examining the timeline of desktop gpus this is not a reasonable assumption, unless apple planned to update the mac pro gpus only once every 5 or 10 years, or if apple originally planned to release their own desktop gpu chips in time for a mac pro upgrade.
Last edited:
i've routed 12 volt rail power to both gpus and the cpu riser card from the 600 watt backup power supply instead of the main power supply and it still shuts down during the thermal interface test. without any apparent overheating - these boards are barely warm with all the cooling i am doing. when i boot back up into asd after the forced shutdown it doesn't say there was a problem during the previous boot - it gives me no additional feedback! it is like the computer itself is limiting the amount of current going to the cpu.
i am back to my original postulation which is that the cpu riser card's tdp limit is more fundamental then just heat. it can't sustainably push more than 130 watts of power continuously. either that or there is some protection circuitry apple has on the board that needs to be disabled to pass this test.
i am back to my original postulation which is that the cpu riser card's tdp limit is more fundamental then just heat. it can't sustainably push more than 130 watts of power continuously. either that or there is some protection circuitry apple has on the board that needs to be disabled to pass this test.
Some bad news for everyone on the thread. At least preliminary. For Mac Pro (Late 2013) it is looking so far like C602J is driving the internal hard drive. And C602J uses AHCI and does not support NVMe. This would also explain why Apple chose to connect the internal hard drive to legacy PCIe 2.0 lanes, because C602J’s hard drive controller only works with PCIe 2.0. The behavior is undefined trying to drive NVMe through the C602J chipset. Depending on how the hardware is implemented cards like angelshark may also require all hard drives on the carrier board to be AHCI to be fully compatible with the C602J.
It is possible this restriction does not apply to external drives if usb or thunderbolt does a pass-thru from the I/O board directly to the CPU. The GPUs connect directly to the CPU so are not limited by the C602J. The limitation would definitely not apply to a second internal hard drive that connected directly to the CPU in a manner similar to the GPUs (but that would require a hardware mod).
This also explains why firmware updates don’t work when the internal drive is non-AHCI. It also explains why the hard drive portion of ASD tests don’t show up if the internal drive is non-AHCI. It explains why Apple never added explicit NVMe support to the driver, because it might be impossible. It probably explains why iMac Pro has custom hard drive controller hardware, so they can future-proof it instead of being limited to the hard drive support built into the main chipset.
Based on this if I am right then immediately remove your NVMe if using it internally in Mac Pro (Late 2013). Replace with Apple stock or any M.2 hard drive based on AHCI technology (though for non-stock you need a connector adapter). The NVMe will have to be external at least for now. This is the only way I can see to guarantee stability.
Some more work to be 100% confident in this feedback. After waiting a few days I may update the first post of the thread.
It is possible this restriction does not apply to external drives if usb or thunderbolt does a pass-thru from the I/O board directly to the CPU. The GPUs connect directly to the CPU so are not limited by the C602J. The limitation would definitely not apply to a second internal hard drive that connected directly to the CPU in a manner similar to the GPUs (but that would require a hardware mod).
This also explains why firmware updates don’t work when the internal drive is non-AHCI. It also explains why the hard drive portion of ASD tests don’t show up if the internal drive is non-AHCI. It explains why Apple never added explicit NVMe support to the driver, because it might be impossible. It probably explains why iMac Pro has custom hard drive controller hardware, so they can future-proof it instead of being limited to the hard drive support built into the main chipset.
Based on this if I am right then immediately remove your NVMe if using it internally in Mac Pro (Late 2013). Replace with Apple stock or any M.2 hard drive based on AHCI technology (though for non-stock you need a connector adapter). The NVMe will have to be external at least for now. This is the only way I can see to guarantee stability.
Some more work to be 100% confident in this feedback. After waiting a few days I may update the first post of the thread.
Last edited:
So I think technically if AHCI Enable bit (of GHC) is cleared (set to false) the hard drive controller in the main chipset is bypassed and the host is free to communicate directly with the drive using any protocol they want. So internal drive might be ok with NVMe if it can be recognized very early on in the boot process at a super low level.
The Mac Pro (Late 2013) seems to work ok with NVMe external drives but I'm not sure what is going on under the hood, how much it prefers ACHI over NVMe. I'd have to do more testing.
The iMac Pro's hardware includes dual NVMe controllers and further acceleration of host-side NVMe on the T2 chip. The iMac Pro defaults to NVMe versus Mac Pro defaulting to ACHI and having to bypass it or otherwise force it to NVMe.
The problem with the specific language of the chipset documentation is a bias that the only purpose of bypassing the AHCI protocol is assumed to be for using a more inferior protocol not a better one aka ‘legacy mechanism’. The people who wrote this doc were willfully ignorant of the future (and if you know the history of Intel this is not surprising for a 2013 product which is a few years before ARM chips started to bypass Intel chips on the low-end and more recently at the mid-range). So there is a chance of electrical limitations of the chipset bypass mechanism, I mean the bypass was clearly never tested for NVMe by Intel otherwise the doc would be more optimistic of the agenda for someone using the bypass.
The Mac Pro (Late 2013) seems to work ok with NVMe external drives but I'm not sure what is going on under the hood, how much it prefers ACHI over NVMe. I'd have to do more testing.
The iMac Pro's hardware includes dual NVMe controllers and further acceleration of host-side NVMe on the T2 chip. The iMac Pro defaults to NVMe versus Mac Pro defaulting to ACHI and having to bypass it or otherwise force it to NVMe.
The problem with the specific language of the chipset documentation is a bias that the only purpose of bypassing the AHCI protocol is assumed to be for using a more inferior protocol not a better one aka ‘legacy mechanism’. The people who wrote this doc were willfully ignorant of the future (and if you know the history of Intel this is not surprising for a 2013 product which is a few years before ARM chips started to bypass Intel chips on the low-end and more recently at the mid-range). So there is a chance of electrical limitations of the chipset bypass mechanism, I mean the bypass was clearly never tested for NVMe by Intel otherwise the doc would be more optimistic of the agenda for someone using the bypass.
Last edited:
I've definitely isolated the 8 differential pairs (4 rx pairs, 4 tx pairs) for the 4 PCIe 2.0 lanes which should be for the hard drive. They are going straight from the CPU to the C602J chipset. I'll post some pics later. I haven't yet isolated the lanes from the hard drive side I need to know those lanes are also going straight to the chipset with no duplicate routing for bypass before I know for sure that the chipset might be the limiting factor when it comes to the internal drive's protocol. I also need to map the pinout for the chipset.
I think I have several variants of C60X/C60XJ in my possession I should double check which flavor is on the logic board ... I double checked my current Mac Pro (Late 2013) Logic Board chipset and yeah it's C602J.
Update: There seems to be alternate confirmation that the PCIe lanes between the CPU and internal SSD go through the main chipset and without an external bypass. I'll still post pics of the flow in the layers once I finish coloring it in.
I think I have several variants of C60X/C60XJ in my possession I should double check which flavor is on the logic board ... I double checked my current Mac Pro (Late 2013) Logic Board chipset and yeah it's C602J.
Update: There seems to be alternate confirmation that the PCIe lanes between the CPU and internal SSD go through the main chipset and without an external bypass. I'll still post pics of the flow in the layers once I finish coloring it in.
Last edited: