NVMe with ST-NGFF2013-C; Vega Internal GPU; Mac Pro 2013 (6,1)

[CodeJingle]

Update: Beginning of thread is mostly NVME related. Work for replacing internal GPU with a 3rd-party GPU starts later in the thread (in progress). The thread is still active. Questions and comments for both NVME and GPU are welcome.

Update: On Mac Pro (late 2013) don't mix APFS and NTFS on same NVMe drive. This will cause frequent and random kernel panics if also using Apple's first-party NTFS driver! Third-party NTFS driver such as by Microsoft/Paragon will still cause kernel panic but only during sleep and shutdown. At least a couple people have seen this (thanks to Kris Kelvin for help narrowing down the issue and for a solid repro). Bug already filed but since NVMe on Mac Pro (late 2013) is third-party hardware only, there is no guarantee the issue will be fixed, unless it can be reproduced on first-party NVMe hardware (iMac Pro or Macbook Pro 2018).

Update: Unedited feedback from Kris Kelvin

• Not all NVMe drives are compatible. Unfortunately, there's no definitive list. Samsung's 960 series seems to work for everyone. Here is a report of someone using a Toshiba XG3.
• The speed will be limited to how many PCIe lanes your Mac supports. Here's an overview over different Models.
• High Sierra must previously have been installed on this Mac (on an AHCI drive). Installing High Sierra upgrades the Mac's EFI Firmware / Boot ROM, allowing for NVMe drives to work. The latest Boot ROM for MacPro6,1 is MP61.0120.B00.
• Your adapter looks to be the Chenyang/CableCC one. Several people reported problems with it. The ones from Sintech and PC PARTS 239 seem to be working properly (including sleep, reboot etc.).
• Make sure the adapter is properly seated and aligned. If it isn't, it might work with fever PCIe lanes (= slower) or not at all.
• Taping the adapter with EM shielding (e.g. Kapton tape) is recommended to prevent EM leaks / shorts.
• High Sierra (or later) is required when using a NVMe SSD. If you're using Internet Recovery, make sure you boot with Option-Command-R (and not just Command-R), so the latest compatible version of macOS is loaded. Bootable installers on USB also work.
• A heat sink for the SSD is not required, but might help sustain performance when stressing it over a long time. This one fits: EK-M.2
• High Sierra's Disk Utility may not show uninitialized devices. If your SSD isn't shown, try this:
  1. Open Disk Utility
  2. Click the top left button in the toolbar ("View") and select "Show All Devices"
  3. Relaunch Disk Utility

Update: Unedited uncut video proof
https://www.dropbox.com/s/emb558u38whfdmm/20171202_000342.mp4?dl=0

I have two questions. I see no thread about NVMe on the trash can Mac Pro, so I made a new thread.

1 - Does Mac Pro 6, 1 (2013) support NVMe drive type and if not is there a firmware hack? I know iMac Pro coming in December is supporting NVMe, but that support may not trickle down to the trash can Mac Pro. Update; it does work. Internal drive in the Mac Pro 2013. Booting even!

2 - Apple's proprietary connector for M.2 is double height with an extra row that is two solid pads on one side. Does anyone know where to get an M.2 adapter dongle that supports this double-height connector format? Update; the two solid pads are connected to ground.

I have Mac Pro 2013 and want to upgrade my M.2 from 1 TB to 2 TB. The only 2 TB M.2 stick on the market besides the OWC custom solution is Samsung MZ-V6P2T0BW. So I got this stick and a dongle to convert from standard M.2 NVMe connector type to Apple's proprietary M.2 connector type. But the dongle connector does not have the second row of pins like my Mac Pro's built-in SSD connector. I've tested it, and it currently does not work in the slot, I'm not sure if the problem is a missing driver or because of the connector difference. Update; it does work. But don't forget to sand down jagged edges using a metal file or sandpaper. And don't forget the tape!

SSD:
https://www.newegg.com/Product/Product.aspx?Item=N82E16820147598

http://www.samsung.com/semiconductor/minisite/ssd/product/consumer/960pro.html

Adapter (USA-only):
Update - PC Parts 239 is discontinued
https://www.amazon.com/gp/product/B01LXUH921

http://www.pcparts239.com

Adapter (inside or outside the USA):
ST-NGFF2013-C (after extensive corroboration it must be the 'C' version; do not get the non-C version)
http://eshop.sintech.cn/ngff-m2-pcie-ssd-card-as-2013-2014-2015-macbook-ssd-p-1229.html

Heat Sink (optional):
https://www.amazon.com/gp/product/B073RHHYCM

https://www.ekwb.com/shop/ek-m-2-nvme-heatsink-black

http://www.samsung.com/semiconducto...document/SAMSUNG_Memory_NVMe_Brochure_web.pdf

Original 1 TB Drive:

New Drive:

Adapter:

Adapter with Tape:

Adapter Sanded Down:

(previously titled "NVMe is working; 3rd-Party Internal GPU; Mac Pro 2013 (6,1)", "NVMe is working; Mac Pro 2013 (6,1)", "2 TB NVMe M.2 SSD in Trash Can Mac Pro")

 

[mikeboss]

there's a thread over at the MacBook forums -> Upgrading 2013/2014 Macbook Pro SSD to M.2 NVMe

the most important thing is to install macOS High Sierra first because this will update the firmware of the Mac Pro (Late 2013). afterwards it should recognize NVMe SSDs. a french guy (username "gilles_polysoft") tested NVMe SSDs with various Macs. he also created a compatibility chart -> ssd-upgrade-table-zip.721929

it seems that there are different types of M.2 <-> Apple SSD adapters on the market. the ones I got from china didn't work (SAMSUNG EVO 960 in MacPro6,1). but this is also a topic later in the thread I linked to. maybe I'll order a new adapter some day...

 

[CodeJingle]

It seems that there are different types of M.2 <-> Apple SSD adapters on the market. The ones I got from China didn't work (SAMSUNG EVO 960 in MacPro6,1). But this is also a topic later in the thread I linked to. Maybe I'll order a new adapter someday...

Yes, but I already got an adapter that claims compatibility with Apple's 2013 2014 2015 M.2 connector format, and I am already running High Sierra, and it is still giving me issue. Update; fixed by booting into High Sierra instead of booting into the version of macOS that is loaded by Internet Recovery.

https://www.amazon.com/gp/product/B01LXUH921

The connector isn't seated correctly, it maybe has only slid in 2/3 of the way, but I'm afraid to force it all the way in the slot. I don't want to break the plug. Update; fixed by sanding the edges of the male end of the connector with a metal file.

 

[CodeJingle]

it seems that there are different types of M.2 <-> Apple SSD adapters on the market

Ok, I got it to fit after filing down the edges, seemed there was a manufacturing defect in the adapter. Hopefully, I didn't damage it in the process.

It still isn't recognizing the drive, dealing with that is the next step. Update; fixed by booting into High Sierra instead of booting into the version of macOS that is loaded by Internet Recovery.

 

[MarkJames68]

Thanks for trying this out and sharing your experience. I may do this upgrade but appreciate you are the guinea pig.

 

[MIKX]

The dedicates Apple M.2 adaptors for MacBooks & MacBook Pro's need a special adaptor to work in PC / Mac Pro M.2 PCI2 adaptors.

I have no idea about 6,1 Mac Pro's M.2 socket type.

There is a Canadian business which sells these but the name escapes me.

Anyone know which company I'm talking about ?

 

[mikeboss]

I have a new SINTECH adapter on order. will let you know how it went as soon as it has arrived...

http://eshop.sintech.cn/ngff-m2-pcie-ssd-card-as-2013-2014-2015-macbook-ssd-p-1139.html
[doublepost=1510134465][/doublepost]

It still isn't recognizing the drive, dealing with that is the next step.

how did you check? System Profiler? Disk Utility? terminal?

 

[CodeJingle]

Anyone knows which company I'm talking about?

The proprietary Apple M.2 connector is the same for the Mac Pro (2013), and the MacBook Pro, for late 2013 or newer. OtherWorldComputing (OWC) aka MacSales sells custom board M.2 upgrades for various macs including Mac Pro (2013). The SSD Mac Pro upgrades available from OWC are slower and cost more, though I bought their external drive enclosure.

I have a new SINTECH adapter on order. will let you know how it went as soon as it has arrived...

How did you check? System Profiler? Disk Utility? Terminal?

From the looks of it, I have an adapter like the one you ordered (quasi-twin though not precisely the same).

https://www.amazon.com/gp/product/B01LXUH921

My external drive housing to use the old SSD has not arrived yet, so the easiest option to test the drive is in Disk Utility from within Internet Recovery. I think the reason this did not work is the on-demand version of macOS retrieved by Internet Recovery is older than High Sierra. Once I get the external drive housing, I can boot from High Sierra normally and test again if the new drive is recognized. Update; yes following this process does lead to the new drive being properly recognized.

https://eshop.macsales.com/item/OWC/MAU3ENPRPCIO/

Regardless I would like to understand both connectors more. I've done a two-sided pin count for both.

Apple's connector (proprietary late 2013 or newer) has a total of 55 (27 + 28) contact points.

Samsung's connector (standard NVMe) has a total of 67 (33 + 34) contact points.

Now, this is just the stupid count. It is obvious from the trace that groups of pins share the same electrical connection, which is standard for PCIe connectors. Tomorrow I will do continuity testing to count the number of unique connections across all contact points, which I am hoping is the same count for both connectors. If so it should be straightforward to fix a possible failure of the adapter. Update; the adapter seems to be functioning properly.

 

[mikeboss]

according to this post #338 the latest sintech adapters have a different layout.

 

[CodeJingle]

according to this post #338 the latest sintech adapters have a different layout.

I received my adapter yesterday; I purchased it four days ago. It has a black coating on the trace, so it is neither adapter mentioned in your link which both have a green coating on the trace. Though if I were to pick, besides the difference in the color of the trace coating, the shape and layout look like the adapter on the right. You can follow my link for comparison.

https://www.amazon.com/gp/product/B01LXUH921

I inspected under the microscope the adapter quality seems good. Like the description of the board on the right, I had trouble fitting the adapter in; the fit was tight. I filed it down a bit then it fit snug like the original apple board.

 

[CodeJingle]

I am getting a count of at most 28 individual signals across all pins for both connectors. So only 28 of the pins are relevant, maybe a couple of extra pins to carry power and for ground. I made a few assumptions since most of the results are based on continuity testing, it might have some issue. This is a sanity check of sorts.

Note the id order is arbitrary, so the id from 1 to 28 does NOT match between the two images (except ground is id 1 for both). That is the next step, to figure out the unique pin mapping together between the two connectors. If it does not add up to at least 28 or results of continuity testing do not match with previous results then there is most likely a problem. Worst case I can draw up the layout for a custom adapter and have a few samples made at a nearby PCB fabrication house, for now, I'm not expecting the need for it. Update; the adapter does work.

 

[CodeJingle]

It looks like the adapter and the SSD are working. I will still update later with the matching pins between connectors. I haven't tested booting from the drive yet. More to come.

Again the link to buy the adapter I am using. If you are in America get it in just a few days with shipping that is low cost or free (with Amazon Prime). Good price too. Note I filed the edges of the adapter down a bit otherwise the fit was too tight. So only buy it if you don't mind that sort of thing.

https://www.amazon.com/gp/product/B01LXUH921

 

[jclmavg]

Very interesting. Is it stable? What's the speed increase?

 

[MarkJames68]

Wow - this is awesome. Can you please post some Blackmagic benchmarks? Also, is the temperature ok without a heat shield?

I spent the $13 for the adapter in anticipation of good news...

 

[ActionableMango]

according to this post #338 the latest sintech adapters have a different layout.

Wow, those adapters sound horrible. Hopefully OP's adapter from Amazon works better than those. It would be a boon to use standard M.2 devices.

 

[jclmavg]

https://www.amazon.com/gp/product/B01LXUH921

Hmm, can't find this adapter for sale in Europe.

 

[CodeJingle]

[doublepost=1510354855][/doublepost]

Hmm, can't find this adapter for sale in Europe.

Here is a screenshot of the adapter if it may help you locate it for sale outside of the Amazon US online store.

[doublepost=1510354966][/doublepost]

Wow - this is awesome. Can you please post some Blackmagic benchmarks? Also, is the temperature ok without a heat shield?

I spent the $13 for the adapter in anticipation of good news...

I will be testing with a heat sink. I hear gossip that Samsung Pro NVMe is known to get hot. The original Apple drive has a heat sink. Update; Blackmagic benchmarks later in the thread.

https://www.amazon.com/gp/product/B073RHHYCM

 

[jclmavg]

Here is a screenshot of the adapter if it may help you locate it for sale outside of the Amazon US online store.

Difficult to find still. Based on the Macbook Pro thread where these adapters are also discussed it seems the Sintech adapter is the one to go for, so bought that one.

You think a heatsink will be necessary for the regular EVO as well?

 

[CodeJingle]

Difficult to find still. Based on the MacBook Pro thread where these adapters are also discussed it seems the Sintech adapter is the one to go for, so bought that one.

You think a heatsink will be necessary for the regular EVO as well?

Sure please use any adapter. The adapter I purchased is only a good choice to buy inside the US. Unfortunately, I cannot provide feedback on any other adapter.

The M.2 drive that comes with the Mac Pro 2013 has a heatsink, so that is one reason for using a heatsink. If EVO is not based on PCIe, then I might not use a heatsink. I would check the comments of purchasers, and the recommendations of the manufacturer.

Here is Samsung feedback

4) Do I need a heat sink for optimal performance of the Samsung NVMe M.2 SSD?

Samsung NVMe SSDs rarely require a heat sink, yet maintain optimal performance thanks to Samsung’s advanced design and manufacturing capabilities coupled with an integrated, heat dissipating thermal label. In rare instances of extreme workloads (transfer of several hundreds of GB), performance may be throttled through Samsung’s dynamic thermal throttling algorithm to protect the SSDs integrity.

Please be aware, however, that an external heat source such as a high- powered graphics card, or CPU placed in close proximity to a Samsung NVMe SSD, can adversely affect air flow around a Samsung NVMe SSD which may lead to performance degradation. Therefore, please ensure there is appropriate air flow around your Samsung NVMe SSD.​

http://downloadcenter.samsung.com/c...11307031/SAMSUNG_Memory_NVMe_Brochure_web.pdf

 

[MarkJames68]

I ordered the parts and will try and give it a test by Monday. Assuming I’m successful I will post some Blackmagic benchmarks.

 

[CodeJingle]

Wow - this is awesome. Can you please post some Blackmagic benchmarks?

Booting from the drive is working. Both drives are native APFS without encryption running macOS High Sierra 10.13.2 Beta (17C67b). Here is the result of Blackmagic Disk Speed Test for both drives. I have the high-end configuration for the Mac Pro 2013 so your results may differ.

Apple original 1 terabyte

Samsung NVMe SSD 960 PRO 2 terabyte

It is kind of sad in 2007 I had 4 SAS 15K drives in hardware RAID0 and was getting 1200 MB/s for a large file copy operation (source and destination the same so actual throughput was double). We haven't come very far in 10 years for sustained throughput. Maybe for random access things have gotten better.

This is hilarious because all I wanted was more space I was not trying to increase speed. There were only two options on the market larger than 1 TB and I chose the cheapest. It is noteworthy to have an incidental 50% increase in disk speed. I expect the higher throughput is now limited by bandwidth of the physical PCIe 2.0 x 4 connection, roughly 1.5 GB/s as an actual empirical limit.

Samsung's specific 2 TB NVMe is only populated on one side yet no 4 TB for sale. I am confused by this, having seen other Samsung NVMe populated on both sides. My main OS drive in 2010 was 1.5 TB when I replaced my optical drive with a second hard disk and had two 750 GB platters in RAID0 on my MacBook Pro. Seven years later I am finally upgrading past 1.5 TB to 2 TB.

 

[MarkJames68]

Booting from the drive is working. Both drives are native APFS without encryption running macOS High Sierra 10.13.2 Beta (17C67b). Here is the result of Blackmagic Disk Speed Test for both drives. I have the high-end configuration for the Mac Pro 2013 so your results may differ.

Apple original 1 terabyte
View attachment 734510
Samsung NVMe SSD 960 PRO 2 terabyte
View attachment 734509
It is kind of sad in 2007 I had 4 SAS 15K drives in hardware RAID0 and was getting 1200 MB/s for a large file copy operation (source and destination the same so actual throughput was double). We haven't come very far in 10 years for sustained throughput. Maybe for random access things have gotten better.

This is hilarious because all I wanted was more space I was not trying to increase speed. There were only two options on the market larger than 1 TB and I chose the cheapest. It is noteworthy to have an incidental 50% increase in disk speed. I expect the higher throughput is now limited by bandwidth of the physical PCIe x 4 connection, roughly 1.5 GB/s as an actual empirical limit.

Thanks for sharing, exactly what I was expecting.

The limit for the nMP is the PCIe 2.0 x4 interface. The drive is capable of 2x the performance under PCIe 3.0. But what you’re seeing is in line with the SSUBX drives (your original drive is a SSUAX which is PCIe 2.0 x2).

 

[CodeJingle]

Just like with the standard M.2 drive, we got it to work, bypassing Apple's proprietary connector, and we can do the same for the graphics cards.

If anyone knows where to get schematics of the Mac Pro 2013 then send me a private message. Otherwise it could take months to map the pins of the two flex cables connecting the graphics cards and M.2 drive to the main hub board (each one is ~300 pins). The problem is made worse by PCIe being a non-public standard and SIG membership is very expensive.

The community wants to eventually be able to replace the internal graphics cards at full x16 bandwidth (or use PCIe for whatever they want), this IS possible. It should not be a big deal to buy a Mac Pro 2013 and transfer the guts into a standard tower case. It should be ethical, moral, legal, etc. But we first have to map the pins on the flex cables and convert them to non-proprietary PCIe connectors. With the flex cables having so many pins some starting documentation is essential.

The minimum goal is simply to show how to boot a Mac Pro 2013 with your own pair of graphics cards plugged in, internally at full x16 bandwidth not externally at x4 bandwidth, directly replacing the two internal cards for GPU purposes (most likely with dual Vega cards). It is easy to support any required passthrough for non-graphics hardware hosted on the original dual GPU boards (so a portion of the D300/D500/D700 circuitry may still be necessary but not for the purposes of GPU). Even if it requires removing the lid and keeping the lid off to show that it works, that would still be a great starting point for others to build off of something that I haven't seen done yet. I will share the repro steps with everyone if I figure it out. But I need some info to get started. If no one has anything to help move me along then I'll have to shelf this goal for now. I am a senior embedded developer by trade I work close to the metal (for example I've coded processor bootstrap a couple times), so this is within my field of expertise.

 

[krakman]

Even if it requires removing the lid and keeping the lid off to show that it works, that would still be a great starting point for others to build off of something that I haven't seen done yet

My nMP died recently and i had a go at taking it apart. I spent a few days trawling the internet for info and this is what if found:

1) i was unable to find a Apple service manual for the machine

2) what little info there is can be found on ifixit

3) Whilst it is safe to dismantle the computer when it is not connected to the mains, you cannot power it up without the outer cover fitted correctly. Whilst you can use a magnet held in a certain area to trip a switch to bypass this safety feature it is not recommended because apparently the internal design uses parts of metal structure to pass electrical current around the machine, therefore this procedure is considered dangerous.

I thank you for your write up about fitting a NMVE SSD to the nMP. I am planning to upgrade my nMP from 256GB to 2TB asap!

 

[CodeJingle]

this is what I found:

1) I was unable to find a Apple service manual for the machine

2) What little info there is can be found on ifixit

3) the internal design uses parts of metal structure to pass electrical current around the machine, therefore this procedure is considered dangerous.

I thank you for your write up about fitting a NMVE SSD to the nMP.

1) This is the first Apple computer that has an entirely 1st party chain of custody (arguably though not technically). It doesn't leak the same as a product made at a production plant out of Shenzhen or Guangdong. Although we will see if more Apple products are assembled in America if it really makes a difference for security. Apple says they have been successful to lower the frequency of leaks coming out of the China supply chain. Update: Based on follow-up in the thread, turns out at least one person has in their possession an Apple service manual [for Mac Pro 2013]. Which makes my response to the first point moot.

2) In the same vein as the previous answer, I can find schematics for nearly every other Apple computer. And I can also find schematics for most iPhone models. But not Mac Pro 2013. Quite infuriating. It isn't a locked-down device it is supposed to be a computer that puts control back in the user's hands. I fully upgraded my Mac Pro 2013 in the Apple store when I bought it in December 2013. And for all that money I can't get minimum documentation of the pinout for the graphics board flex cable.

3) Let me clarify the 'how electricity works' bit. The metal housing of the Mac Pro 2013, including the external cover, act as both a heat sink and as 'ground.' I don't think any portion of the solid metal housing is used for transferring active power it is exclusive for grounding purposes. The extra electrical charge that has left the circuits and is floating in the metal housing is pulled toward earth ground, the third prong in your AC outlet. Without proper grounding, the extra charge will continue to flow around the housing like a faraday cage building up more charge and eventually will arc and discharge as static electricity, either outside or back into a circuit inside the housing. In the case of internal static discharge, it will continue to damage the computer until a board frys, or you switch to an outlet with proper grounding. Even if it never arcs, buildup from a failure to leak to ground will still cause the flow of electricity to eventually stop and then the computer will start failing and then kernel panic. I recommend always connecting the Mac Pro 2013 through a UPS, since most of them check for failure to ground and will refuse to operate until the ground fault is fixed. I have damaged a Mac Pro 2013 previously due to an unknown ground fault.

Legacy PC-esque towers also have large metal plates inside the casing and usually, the slide-open access door is also made of metal. It is easy enough to sprawl out all the components of a disassembled Mac Pro 2013 removed from its original metal housing and insert those components into a legacy tower with all components mounted to at least be touching one of metal walls of the casing. The stray charge will get pulled toward the outlet and drained to earth ground.

How NOT to re-build your Mac Pro 2013 after removing it from the case
http://www.instructables.com/id/Standing-PC-Case/

Picture of unpopulated metal tower (hint: all metal; made to be well grounded aka anti-static)

Picture of a populated metal tower (hint: all metal; all installed components have at least one bare metal surface in contact with the bare metal of the housing.

Bypassing the "don't turn on your computer when the lid is off" is pretty simple. It is also easy to re-ground all of the components after removing them from the original casing. Those are baby steps.

Compare that to the effort and time required for using a razor blade to gently scrape off the top and bottom layer trace coating of both working graphics cards without destroying them. To create a separate ~300-pin breakout board for the two flex cables. The time just to build one ~300 pin breakout board by hand is a whole day, ~8 hours straight for a build of high enough quality that the Mac Pro 2013 still works in pass-thru. Then buy and use several expensive logic probes fast enough to handle the full bandwidth of PCIe 3.0. Update - continuity testing from the unpopulated cpu socket may work to diagram the connector pins and is non-destructive.