Mac Pro (Late 2013) - Amfeltec AngelShark Carrier Board - Storage Upgrade

[mikas]

Separate drives, RAID, or any combination, but the speed limit is hit at 1500 MBps (x4 PCIe 2.0)

AngelShark M.2 Carrier inside 2013 Mac Pro

real world speed test results for performance minded Macintosh users

barefeats.com

While a RAID might give some other benefits, there will be no more continuous speed than that, and with separate drives it's the same. Copying from one drive to another is limited by that same bottleneck too. That's how I understand it. And I believe the lanes are shared for some other uses too through Platform Contoller Hub.

I bought this blade to have more storage inside that black metal tube, not got speed increase.

 

[joevt]

Separate drives, RAID, or any combination, but the speed limit is hit at 1500 MBps (x4 PCIe 2.0)

Maybe a raid of three Thunderbolt devices connected to three thunderbolt buses can give up to 4500 MB/s?

 

[mikas]

Maybe so. At least in theory you could achieve even more maybe.
I've been reading some articles lately:

The Mac Pro Review (Late 2013)

www.anandtech.com

and

Thunderbolt (interface) - Wikipedia

en.wikipedia.org

I found it interesting you actually CAN use and buy optical cables with Apple TB 2 to 3 adapters, and span the distance far more than with usual TB copper cables. They are not cheap though (in hundreds of $¢€ /10m or 20m).

As of March 2020 there were no optical Thunderbolt 3 cables on the market. However, optical Thunderbolt 1 and 2 cables can be used with Apple's Thunderbolt 3 (USB-C) to Thunderbolt 2 adapters on each end of the cable. This achieves connections up to the 60 m (200 ft) maximum offered by previous generations of the standard.

I find it interesting - but that's totally off topic here.

 

[joevt]

Maybe so. At least in theory you could achieve even more maybe.
I've been reading some articles lately:

The Mac Pro Review (Late 2013)

www.anandtech.com

and

Thunderbolt (interface) - Wikipedia

en.wikipedia.org

Well, there are only 3 busses, each limited to 16 Gbps, and the writer of the anadtech article says he got 1.38GB/s which is 11 Gbps. So the max is maybe 4140 MB/s. I used a more ideal 1500 MB/s per drive (12 Gbps) for my guess.

I found it interesting you actually CAN use and buy optical cables with Apple TB 2 to 3 adapters, and span the distance far more than with usual TB copper cables. They are not cheap though (in hundreds of $¢€ /10m or 20m).

There exists now Thunderbolt 3 optical cables so you don't need to use a Thunderbolt 2 optical cable with Thunderbolt 3 adapters and they allow 40 Gbps instead of 20 Gbps.
https://www.corning.com/optical-cab...e/en/products/thunderbolt-optical-cables.html

 

[mikas]

Yes that's right, I realized that too after reading. Thunderbolt 2 spec is 20 Gbps, but there are 4 PCIe 2.0 lanes per TB controller cutting the limit to 16 Gbps. If I understood it all right.
But now that I will put Angel Shark in there, and a couple of fast NVMes in it, I could have a boot volume and stripe one or two NVMe drives with those three thunderbolt drives. That would (in theory) make it ~5500 MBps, wouldn't it?

Probably not ideal, I reckon, but just for thoughts. And the lowest thunderbolt ports seem to share lanes with HDMI, which I am using too, so that eats the bandwith too from that above imagined RAID setup (which I am not going to build anyway )

I don't think I have seen or touched thunderbolt optical cables. Are they thick and stiff, like compared to optical audio cables?

 

[joevt]

Yes that's right, I realized that too after reading. Thunderbolt 2 spec is 20 Gbps, but there are 4 PCIe 2.0 lanes per TB controller cutting the limit to 16 Gbps. If I understood it all right.
But now that I will put Angel Shark in there, and a couple of fast NVMes in it, I could have a boot volume and stripe one or two NVMe drives with those three thunderbolt drives. That would (in theory) make it ~5500 MBps, wouldn't it?

Yes. 4500+1500 = 6000 MB/s. 5500 is probably more reasonable.

And the lowest thunderbolt ports seem to share lanes with HDMI, which I am using too, so that eats the bandwith too from that above imagined RAID setup (which I am not going to build anyway )

HDMI is not connected to Thunderbolt PCIe or display bandwidth. You can see in the Mac Pro 2013 block diagram that the HDMI port comes directly from one of the six GPU outputs. There is a MUX to switch that output between HDMI and one of the Thunderbolt controllers. The RAID only uses about 3 of the 6 Thunderbolt ports - you won't loose performance using any of the unused Thunderbolt ports for a display because DisplayPort traffic on one port of a bus does not affect PCIe traffic on the other port of the same bus.

https://egpu.io/wp-content/uploads/2017/12/late-2013-mac-pro-system-block-diagram.png

I don't think I have seen or touched thunderbolt optical cables. Are they thick and stiff, like compared to optical audio cables?

I think optical cable is thin. Copper cables are thick to reduce resistance which is not necessary for optical. Optical has larger ends for the extra electronics required to convert electrical to optical. You can see pictures of Thunderbolt optical cables.

 

[mikas]

Thanks for the pic. I've been studying it.

 

[SamPotts]

Does anyone know if this can be used for 3 seperate drives or are there speed penalties?

I used the three drives separately. I don't think you'd get full speed from all three if you're accessing them all at once but for me, that would have been rare anyway and it was more than fast enough for my needs.

Edit: I was way too slow

 

[Tastannin]

If anyone is still looking for these, I have two AngelShark carrier boards for sale as a bundle posted in the Marketplace.

 

[Spacedust]

I got an issue with these. When paired with standard Apple 1 TB SSD + Intel Optane 905p 380 GB + Intel 760p 2 TB when you click EFI boot icon to boot Windows Mac Pro simply turns off.

If you remove one of the Intel drives you can boot Windows normally.

Under macOS all works well. Both Intel drives are visible and usable. What's going on ?

 

[Spacedust]

Update: I've tried selecting Windows BootCamp NVMe drive from Big Sur and rebooting. Voila!

Windows 10 booted without issues, but at first boot second NVMe controller was showing code 12 error and secondary NVMe was absent.

Post reboot all went to normal and works well.

The only issue is you can't click EFI boot icon directly or Mac Pro will simply turn off.

Update: It's now fixed. There was some old EFI partition on other macOS drive containing Windows files.

 

[thatboyisgenius]

If anyone is still looking for these, I have two AngelShark carrier boards for sale as a bundle posted in the Marketplace.

yes I'm interested in a carrier board... are they in working condition?

 

[BartConway]

If anyone is still looking for these, I have two AngelShark carrier boards for sale as a bundle posted in the Marketplace.

Count me in as interested in picking one of these up (after exhausting my options trying to successfully run external m.2 drives through the onboard T2 ports, via Apple's expensive adapters... they won't). If one of the carrier boards is unsold, please contact me.

 

[joevt]

Count me in as interested in picking one of these up (after exhausting my options trying to successfully run external m.2 drives through the onboard T2 ports, via Apple's expensive adapters... they won't). If one of the carrier boards is unsold, please contact me.

If the M.2 drives are bus powered then you need to connect them to a powered device like a Thunderbolt dock. I know you said you had problems with a TS3+ Thunderbolt dock. Maybe a different dock would work.

Anyway, here's the marketplace: https://forums.macrumors.com/thread...lled-700.2284290/?post=29599984#post-29599984

 

[BartConway]

If the M.2 drives are bus powered then you need to connect them to a powered device like a Thunderbolt dock. I know you said you had problems with a TS3+ Thunderbolt dock. Maybe a different dock would work.

Indeed... I learned that the hard way, hence picking up that TS3+. I would sure value input from anyone else here if they are successfully running multiple m.2 SSDs externally on the 2013 MP at anything approaching reasonable speed (by which I would mean, in the 800-900 MB/s or higher range on R/W) through their Thunderbolt 2 ports.

I would rather not burn through multiple expensive powered docks, when they are all seemingly rated at the same essential power outputs, to take a chance after the well-rated CalDigit failed so miserably. I've always had great success on my 2012 MP tower running a series of external m.2 drives just fine and dandy via PCI cards, so I find it kind of incredible the same thing can't be done on the 2013 over Thunderbolt... even WITH a powered dock.

 

[tythomas]

If anyone is still looking for these, I have two AngelShark carrier boards for sale as a bundle posted in the Marketplace.

Still for sale?

 

[Antonio Camara]

Do somebody have one? I'm looking after it...

 

[Macstefws]

Tastannin said: If anyone is still looking for these, I have two AngelShark carrier boards for sale as a bundle posted in the Marketplace.

Still for sale? Somebody else might have an AngelShark Carrier Board for sale?

 

[Spacedust]

Guys - do you have such buzzing when Mac Pro is turned off ? Turn up the volume please.

 

[grouch]

I've been thinking about using my 2013 mac pro as a home server but the single drive kinda dissuades me a bit, this would be a great solution if they weren't so expensive and I could even find them.

 

[PresidentRoss]

Test Mac Pro Specifications
MacPro6,1
BootROM: MP61.0120.B00
macOS High Sierra 10.13.1 (17B1003)
E5-1660v2 3.7GHz (15MB L3) Hexacore
16GB of Apple/SKHynix PC3-14900 ECC
Apple/Samsung SM0256G PCIe SSD (SSUBX based)
2x AMD FirePro D300 with 2GB VRAM

Testing Tools for SSD Benchmarks
AJA System Test Lite - Primary Test - 4GB File Size
BlackMagic Disk Speed Test (Outdated, but still a decent tool to check against)

http://amfeltec.com/products/mac-pro-late-2013-carrier-board-for-m-2-pcie-ssd-modules/

I'm working on getting one in to test. Really interesting how they power the board from the Bus Bars that go to the GPU. They are using the two black screws in the upper right hand corner of Graphics Board B. They include two standoffs. Pretty cool!

UPDATE 1/16/18 12:00pm CST

Added pictures to post.

10.13.2 didn't change the compatibility of any drives at the bottom of this post.

The Samsung 960 EVO and 960 PRO are the best available to use with this adapter board. Because of the internal link speed, you will not see any increase in speed with the 960 EVO or 960 PRO, but you will be able to get up to 6TB of internal storage, which is nice.

The other interesting thing is the board has boot LEDs so that you are aware that it is working probably. These do not blink or stay on once the system has done its NVRAM and POST checks.

https://photos.google.com/u/2/photo/AF1QipMkoDZcx4TGyFJqZa3GGSbnvoJjNMGFwqezgHo2

UPDATE 12/1/17 12:30pm CST
I have received the card and I'm currently taking pictures and documenting benchmarks on difference NVMe drives.

UPDATE 12/2/17 10:30am CST
Pictures of AngelShark Board installed. It's a rather well engineered board. It is rather thin (looks to be a three or four layer board) and kinda flexible.

UPDATE 12/3/17 10:30am CST
I've been testing the board for multiple days and I'm still experiencing slow write speeds on any and all compatible m.2 SSDs. 300MB/s is the fastest I've been able to achieve, on BlackMagic Disk Speed Test and AJA System Test Lite.

Internal Apple SSD shown as external drive within Disk utility.

UPDATE 12/3/17 11:15am CST
300MB/s writes are still present even after a fresh install of High Sierra on the PM951 as the boot drive. Apple SM0256G speeds are the same within the boot environment on the PM951, 1400R/1200W.

UPDATE 12/3/17 5:30pm CST
Ran out and grabbed a couple of Samsung 960 EVO 256GB. They show full bandwidth on read and write. They are only performing at PCIe 2.0 x4 speed (up to 2GB/s). I RAIDed them in a RAID0 and same result as a single drive. ~1500MB/s READ & ~1350MB/s WRITE.

Single 960 EVO

RAID0 of 2x 256GB 960 EVO

COMPATIBLE
Samsung 960 EVO - Full Bandwidth on Read & Write
Samsung 960 Pro - Full Bandwidth on Read & Write
Samsung PM951 NVMe - Full Bandwidth Read, 1/4 Bandwidth Write
Toshiba XG4 - Full Bandwidth Read, 1/4 Bandwidth Write
Toshiba XG3 - Full Bandwidth Read, 1/4 Bandwidth Write

NOT COMPATIBLE
LiteOn CX2 (commonly used in Dell Latitude & Precision Notebooks)

(List up to date as of December 3rd, 2017 at 5:30pm CST)

Keywords
Mac Pro (Late 2013) Carrier Board for M.2 PCIe SSD Modules
Amfeltec AngelShark Carrier Board
SKU-088-01 AngelShark Carrier Board for M.2 SSD modules (M.2 key M)

I have an issue with mine. Only one of the slots works 100% of the time. The other slot only shows up after restarting 2-3 times. Ive tried swapping the name out with another and tried on multiple OS's such as High Sierra, Mojave, and Catalina. Still an issue. I tried reinstalling the whole thing step by step, same issue. They won't return my calls or respond to my email

 

[PresidentRoss]

@Kris Kelvin I forgot to mention the lights! It's alive! On first power on, it will show blue, red, and green lights if I remember correctly. Glad you got Windows installed and working. I always just used it in a VM for my needs.

What do the different lights mean? Mine shines blue no matter what

 

[RyanXM]

What do the different lights mean? Mine shines blue no matter what

These were never laid out in a manual, so don't have a clue.

 

[PresidentRoss]

These were never laid out in a manual, so don't have a clue.

Figured it out. Each light corresponds to one of the 3 NVMEs. If a light is on a color it means that NVME is detected but there is no connection to the data on the drive to the carrier board

Usually reseating the NVME does not fix it, but swapping the two NVMEs slots does

Lights can also mean that the board isn’t getting enough power, usually caused by having the Mac plugged into a surge protector or a plugin hub. To fix this it needs to be plugged directly into a wall outlet. Some older houses this wont matter since older wire hookups never get enough power anyway

So it’ll need a newer house with modern wiring to fix

Also having too many high power USB devices (external USB hard drives) plugged in can take power away from the NVME slots which can cause one of the lights to kick on and the drive to not show up.

 

[onesixright]

Hi All,

Been a while 😀 Resurrecting this one...

I added a second disk. A WD Black SN 770, but it refuses to show up (next to my EVO 970). The EVO works fine.

Any ideas what this can be? I also tried a WD Blue SN 580, same story.

I swapped bot drives (so the EVO and the WD) and the EVO still works. So I assume the port is good.

Could this really be a NVMe compatibly issue? Need another EVO? Or is there more going on? I also removed all disks, only used the WD, also doesn't show. The LEDs only flash on boot, and then nothing.

Some help or someone that can shed some light on this matter will be greatly appreciated!

Thanks!