Highpoint 7101A - PCIe 3.0 SSD performance for the cMP

[flygbuss]

The Sonnet has no CPU - it's just external PCIe chassis, so it has to be connected via TB2 to a Mac Pro. Then that Mac Pro can enable file sharing using 10 GbE. It's also possible to do Thunderbolt networking but I don't think the performance is much better than 10 GbE, or it might be worse, have to run some benchmarks to be sure.

Yes, that’s what I meant. Sorry if I’ve been unclear.
Main purpose would be to use the sonnet + highpoint connected via TB2.
So each Mac get its own fast storage (1300MB/s)
Additionally it would be great to be able to access the storage across the two Macs via 10GbE.

I think I might give it a try and report back how it performs.

thank you.

 

[Adam Chuan]

Introduction

Stumbling into an Amazon warehouse Deal, I picked up the HighPoint 7101a NVMe PCIe adapter, equipped with a MacPro friendly Broadcom 8747 PCIe bridge at a great price. While Highpoint's acknowledgement of MacPro compatibility was sketchy at best, I ordered one to go along with a recently acquired Samsung 970 Pro NVMe Boot drive.


Some Images

PEX 8747
48-Lane, 5-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA
The ExpressLane™ PEX 8747 is a 48-lane, 5-port, PCIe Gen 3 switch device
Technical Specification Document link

On to the results
Let the numbers speak for themselves. I'm VERY impressed with the near PCIe 3.x performance on a '09 CMP with the Highpoint 7101A PCIe SSD Adapter and the Samsung 970 pro. Yes... Read performance has almost doubled from 1500MBs to 3000MBs. Writes are up 50%.

Taking a look under the hood, The 7101A addresses PCIe SSD’s at a pcie 3.0 x4 8.0Gts link, saturating the x16 a pcie 2.0 connection at close to PCIe 3.0 speeds.

If there is a Holy Grail of PCIe SSD adapters for the cMP, this may be the one.

View attachment 767207
970 Pro in a standard m2 x4 adapter with a heatsink.

View attachment 767208 970Pro in the 7101A.
View attachment 767209

Whilst the 7101a usually costs $400, a qty can be found at enthusiast pricing on Amazon for $275.

It’s clear the HighPoint 7101a brings a new level of PCIe NVMe / AHCI Sata Express performance to the 4,1/5,1 that’s unattainable with any other controller.

今天早上，我能夠拆解Amfeltec Squid 2.0x4，刪除了3個SM951及其在軟件RAID中安裝的散熱器。將它們移到7101A上，發現了一些有趣的結果：

Highpoint 7101A一次尋址單個SSD，比Amfeltec Squid更快。從我們老化的Mac Pro中獲得接近PCIe 3.0的速度。

使用Amfeltec Squid的軟件RAID 0設置，Highpoint 7101A比Squid慢25％。也許我需要用不同的塊大小重新格式化數組。


讓我們從BlackMagic Speed Test開始：

最高點7101A 3xSM951
View attachment 767333

Introduction

Stumbling into an Amazon warehouse Deal, I picked up the HighPoint 7101a NVMe PCIe adapter, equipped with a MacPro friendly Broadcom 8747 PCIe bridge at a great price. While Highpoint's acknowledgement of MacPro compatibility was sketchy at best, I ordered one to go along with a recently acquired Samsung 970 Pro NVMe Boot drive.


Some Images

PEX 8747
48-Lane, 5-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA
The ExpressLane™ PEX 8747 is a 48-lane, 5-port, PCIe Gen 3 switch device
Technical Specification Document link

On to the results
Let the numbers speak for themselves. I'm VERY impressed with the near PCIe 3.x performance on a '09 CMP with the Highpoint 7101A PCIe SSD Adapter and the Samsung 970 pro. Yes... Read performance has almost doubled from 1500MBs to 3000MBs. Writes are up 50%.

Taking a look under the hood, The 7101A addresses PCIe SSD’s at a pcie 3.0 x4 8.0Gts link, saturating the x16 a pcie 2.0 connection at close to PCIe 3.0 speeds.

If there is a Holy Grail of PCIe SSD adapters for the cMP, this may be the one.

View attachment 767207
970 Pro in a standard m2 x4 adapter with a heatsink.

View attachment 767208 970Pro in the 7101A.
View attachment 767209

Whilst the 7101a usually costs $400, a qty can be found at enthusiast pricing on Amazon for $275.

It’s clear the HighPoint 7101a brings a new level of PCIe NVMe / AHCI Sata Express performance to the 4,1/5,1 that’s unattainable with any other controller.

I was able to tear down the Amfeltec Squid 2.0x4 this morning, removing 3 SM951's and their heatsinks that are setup in a software raid. Moving them across to the 7101A, some interesting results came to light:

Addressing a single SSD at a time, the Highpoint 7101A is MUCH FASTER than the Amfeltec Squid. Allowing near PCIe 3.0 speeds from our aging mac pro.

Using a Software RAID 0 setup for the Amfeltec Squid, the Highpoint 7101A is about 25% SLOWER than the Squid. Perhaps I need to reformat the array with a different block size.


Let's start with BlackMagic Speed Test:

Highpoint 7101A 3xSM951
View attachment 767333
Highpoint 7101A 2xSM951 ** FASTER THAN 3X SM951 in 7101A**
View attachment 767369
Amfeltec Squid - 3xSM951
View attachment 767336

Amorphous Disk Mark (FREE - GO GET THIS ONE)

Highpoint 7101A 3xSM951
View attachment 767337
Highpoint 7101A 2xSM951
View attachment 767371
Amfeltec Squid - 3xSM951
View attachment 767340
QuickBench is another great test to show bandwith. Lets look first at 4k - 1m tests:

Highpoint 7101A 3x SM951
View attachment 767343
Highpoint 7101A 2x SM951

Amfeltec Squid - 3xSM951
View attachment 767344

Lets look at some larger file transfers.

Highpoint 7101A 3xSM951

Amfeltec Squid - 3xSM951

Hi!
I have a question.

My Mac Pro 5,1 is 2012mid.

Which PCIe Slot did you install the SSD7101A in?

My SSD7101A can’t install in the slot2,it can’t mount the disk and can’t not format.

It works on the slot 1,3,4 are normally, but the disk speed is not ideal.

I had try the other Mac Pro 5,1 2010 and 2012,it’s the same result.

I wish that you can help me.

I use four disks are Samsung 970 Evo Plus.

I want to create the Raid0.

My GPU is Nvidia Titan Blcak for Mac version.

My Mac Pro BootRom is 144.0.0.0.0,and my Samsung firmware is updated to match the 970 too.

 

[handheldgames]

Hi!
I have a question.

My Mac Pro 5,1 is 2012mid.

Which PCIe Slot did you install the SSD7101A in?

My SSD7101A can’t install in the slot2,it can’t mount the disk and can’t not format.

It works on the slot 1,3,4 are normally, but the disk speed is not ideal.

I had try the other Mac Pro 5,1 2010 and 2012,it’s the same result.

I wish that you can help me.

I use four disks are Samsung 970 Evo Plus.

I want to create the Raid0.

My GPU is Nvidia Titan Blcak for Mac version.

My Mac Pro BootRom is 144.0.0.0.0,and my Samsung firmware is updated to match the 970 too.

I'm using slot 1 for the SSD7101a on two different Mac pro's with the video card in slot 2. Although this approach blocks slot #3, you are gaining 4 M.2 slots in the process.

While I can run the SSD7101a in slot #2 and the video card in slot#1, the SSD7101a blocks the fans on the video card.
Speed will max out around 6000 MB/s with 3 SSD's.
Have you considered booting from one of the 970 Evo and a Raid 0 with 3 SSD?

 

[roobarb!]

Speed will max out around 6000 MB/s with 3 SSD's.
Have you considered booting from one of the 970 Evo and a Raid 0 with 3 SSD?

This is definitely the most efficient option for speed and versatility. You won't get faster than 6000 MB/s, so if you've got four 970 Evo Plus blades putting the fourth in RAID 0 only improves the capacity of the array, not the speed when in cached writing mode.

However, if you're wanting to maintain the higest possible write speeds for long periods of time with the Evo Plus blades, there is a benefit to being less efficient. Once the faster cache is exhausted on the Evo Plus your write speeds will drop per blade, almost certainly below the maximum throughput of the 7101A. Good heatsinks and all four in RAID 0 will keep the sustained bandwidth as high as possible.

Your only option beyond that is the 970 Pro, which just won't quit in sustained write performance, but only offers this in 1 TB maximum capacities. Though you would only need three to max out the 7101A's bandwidth.

 

[Adam Chuan]

I'm using slot 1 for the SSD7101a on two different Mac pro's with the video card in slot 2. Although this approach blocks slot #3, you are gaining 4 M.2 slots in the process.

While I can run the SSD7101a in slot #2 and the video card in slot#1, the SSD7101a blocks the fans on the video card.
Speed will max out around 6000 MB/s with 3 SSD's.
Have you considered booting from one of the 970 Evo and a Raid 0 with 3 SSD?

Thank for your reply.

I will try to use 3 SSD to create the raid0.

now,I use the 4 SSD to create the raid0 in slot 1,but the read and write speed can’t up to the 3000.

so it’s very confused to me.

I don’t know what driver or kext I need to do in my Mac Pro 5,1.

Is it create the 4 SSD to raid0 causes this problem?

 

[tsialex]

Thank for your reply.

I will try to use 3 SSD to create the raid0.

now,I use the 4 SSD to create the raid0 in slot 1,but the read and write speed can’t up to the 3000.

so it’s very confused to me.

I don’t know what driver or kext I need to do in my Mac Pro 5,1.

Is it create the 4 SSD to raid0 causes this problem?

You don't need driver or kexts with SSD7101-A at all, MP5,1 firmware fully support it natively. Seems you have a defective card, since it happens with two Mac Pros.

Slot 1 or 2 will get you to ~6200MB/s, with 3 or 4 up to 1500~1550MB/s

 

[Adam Chuan]

You don't need driver or kexts with SSD7101-A at all, MP5,1 firmware fully support it natively. Seems you have a defective card, since it happens with two Mac Pros.

Slot 1 or 2 will get you to ~6200MB/s, with 3 or 4 up to 1500~1550MB/s

Thank you.

I am trying to create the raid0 in 3 SSD.

If the problem still in there.

I will repair my 7101 card.

 

[joevt]

thank you for your reply.

In fact, I tried more than 3 Mac Pros.

These models are all about 2010 to 2012.

All about the problem that slot 2 cannot read the 7101 card.

And I tried to install the 7101 card in the other slots 1, 3, and 4 of these Mac Pros, and they work fine.

I have tried installing GPUs or other tools related to PCI devices in slot 2, and they also work fine.

The problem that occurs on every Mac Pro I have tried is the same.

Now, I created raid0 from 3 SSDs in response to [USER = 301041] [USER = 301041] @handheldgames [/ USER] [/ USER], but the read and write speeds are still the same, it always stops reading 2800Mb / s And write 2400Mb in / s.

I started to suspect that this was related to my 7101 card issue.

Install pciutils and use it to check the PCIe link width and link speed of the NVMe, the 7101 card, and the Mac Pro's slot 2. Or use the pcietree.sh script to check the link width and link speed (requires pciutils).

 

[Adam Chuan]

Install pciutils and use it to check the PCIe link width and link speed of the NVMe, the 7101 card, and the Mac Pro's slot 2. Or use the pcietree.sh script to check the link width and link speed (requires pciutils).

How to use?
And where can I get this?

 

[joevt]

How to use?
And where can I get this?

#223

 

[handheldgames]

Thank you.

I am trying to create the raid0 in 3 SSD.

If the problem still in there.

I will repair my 7101 card.

Do you have an updated firmware in your 5,1? version 144 is the latest.
You can also format your raid 0 with HFS+ for fastest performance

 

[Adam Chuan]

Do you have an updated firmware in your 5,1? version 144 is the latest.
You can also format your raid 0 with HFS+ for fastest performance

Yes, I have updated firmware in my 5,1, and the version is 144.

 

[Adam Chuan]

#223

hi
I have try to use your teach step.
but it's nothing happened in there.
I don't know which step is wrong.
My terminal display that no such file or directory and no command not found.

I download the pcitree.sh,pciutils-3.6.2,directHW.kext and kext drop.

and I don't know the" /usr/local/sbin to the file /etc/paths" how to move or use it.

thanks you so much .

 

[handheldgames]

hi
I have try to use your teach step.
but it's nothing happened in there.
I don't know which step is wrong.
My terminal display that no such file or directory and no command not found.

I download the pcitree.sh,pciutils-3.6.2,directHW.kext and kext drop.

and I don't know the" /usr/local/sbin to the file /etc/paths" how to move or use it.

thanks you so much .
View attachment 869927

View attachment 869929

View attachment 869930

View attachment 869931

View attachment 869932

Thanks for sharing the screenshots. Regarding your 1st image, you should crop the last 3 lines with the serial numbers and re-post. You don't want those numbers getting into circulation.

The 970's evo look correct. Each NVME SSD is a x4 connection running at a PCIe 3.0 8.0 GT/s connection.

What kind of performance are you seeing on you raid running Aja System Test Light?
Link: https://apps.apple.com/us/app/aja-system-test-lite/id1092006274?mt=12

 

[Adam Chuan]

Thanks for sharing the screenshots. Regarding your 1st image, you should crop the last 3 lines with the serial numbers and re-post. You don't want those numbers getting into circulation.

The 970's evo look correct. Each NVME SSD is a x4 connection running at a PCIe 3.0 8.0 GT/s connection.

What kind of performance are you seeing on you raid running Aja System Test Light?
Link: https://apps.apple.com/us/app/aja-system-test-lite/id1092006274?mt=12

It’s the same.
The write and read average still stop at the 2400/2800.

So , I am confused now.

I have sent the email to HighPoint , but they just reply to me the hardware is not match for Mac Pro, because the Mac Pro PCIe slot only support 2.0, 7101 is support PCIe 3.0 .

 

[handheldgames]

It’s the same.
The write and read average still stop at the 2400/2800.

So , I am confused now.

I have sent the email to HighPoint , but they just reply to me the hardware is not match for Mac Pro, because the Mac Pro PCIe slot only support 2.0, 7101 is support PCIe 3.0 .

Hmmm. It sounds like your highpoint is running at x8 and not x16.
Considering it does not work in your 2nd slot, where other devices work, there may be something wrong with your adapter?

How is your raid setup? Can you share the info from disk utility?
I'd suggest deleting the array and starting from scratch with 2 disks and see if there is any difference in performance.

 

[joevt]

I have try to use your teach step.
but it's nothing happened in there.
I don't know which step is wrong.
My terminal display that no such file or directory and no command not found.

I download the pcitree.sh,pciutils-3.6.2,directHW.kext and kext drop.

and I don't know the" /usr/local/sbin to the file /etc/paths" how to move or use it.

"/usr/local/sbin" is a folder that may exist on your system (or it might be created by building pciutils). To view the folder, you can type "ls -l /usr/local/sbin". It should have at least the following files:

-rwxr-xr-x  1 joevt  wheel    99528 30 Jun 14:12 lspci
-rwxr-xr-x  1 joevt  wheel    45744 30 Jun 14:12 setpci
-rwxr-xr-x  1 joevt  wheel     1752 30 Jun 14:12 update-pciids

"/etc/paths" is a file that should exist on your system. To view the file, you can type "cat /etc/paths". It should look something like this:

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/usr/local/sbin

If it doesn't have /usr/local/sbin, then you can add it with the following command:

sudo echo /usr/local/sbin >> /etc/paths

I don't think you need directHW.kext with High Sierra. Check boot-args with "nvram boot-args". It should look something like this:

boot-args    debug=0x144

You can set it with this command:

sudo nvram 'boot-args=debug=0x144'

Test lspci with this:

sudo lspci -G

 

[Adam Chuan]

Hmmm. It sounds like your highpoint is running at x8 and not x16.
Considering it does not work in your 2nd slot, where other devices work, there may be something wrong with your adapter?

How is your raid setup? Can you share the info from disk utility?
I'd suggest deleting the array and starting from scratch with 2 disks and see if there is any difference in performance.

These photos are showing in my disk utility.

And this photo is about the 1 SSD speed in system.

And this speed is about Raid0(3 SSD) in system.

 

[MIKX]

Adam Chuan

Some system files are by default " hidden". or "invisible".

Google "OS X make hidden files visible " for Terminal commands.

Personally, I use Main Menu which has a drop down menu which allows you to " See hidden / invisible files.

Main Menu 3.1.0 is a very useful tool and still works in Mojave

 

[joevt]

These photos are showing in my disk utility.

And this photo is about the 1 SSD speed in system.

And this speed is about Raid0(3 SSD) in system.

None of those screen shots tell us about the PCIe link width and link speed. Use pciutils to find out.

 

[handheldgames]

These photos are showing in my disk utility.

View attachment 870111

View attachment 870112

View attachment 870113

And this photo is about the 1 SSD speed in system.

View attachment 870114

And this speed is about Raid0(3 SSD) in system.

View attachment 870115

As @joevt mentioned, it would be worth while to use pcitools to verify the speed link. Although, based on your screenshots, it looks like you adapter is stuck at an x8 connection.

If you are on a timeline for returning/exchanging the SSD7101-a, I would highly suggest doing so.

Running a quick test this morning with a 2 disk 970Pro raid 0 array, I'm seeing:

• max performance: 5218MB/sec Write and 5645MB/sec Read
• Average performance: 4200MB/s Write and 5475MB/s Read
• Occasional hiccups with APFS: 570MB/s Write and 760MB/s Read*

*occasional slowdowns did not occur with - Mac OS Extended (Journaled)

 

[Adam Chuan]

As @joevt mentioned, it would be worth while to use pcitools to verify the speed link. Although, based on your screenshots, it looks like you adapter is stuck at an x8 connection.

If you are on a timeline for returning/exchanging the SSD7101-a, I would highly suggest doing so.

Running a quick test this morning with a 2 disk 970Pro raid 0 array, I'm seeing:

• max performance: 5218MB/sec Write and 5645MB/sec Read
• Average performance: 4200MB/s Write and 5475MB/s Read
• Occasional hiccups with APFS: 570MB/s Write and 760MB/s Read*

*occasional slowdowns did not occur with - Mac OS Extended (Journaled)

In fact, I don’t know how to use the PCIe tools.

And I don’t know these code where can I write into? terminal? or anywhere?

I am the terminal idiot.

maybe I need some screenshot about the terminal step.

I have try so many times to use the code in the PCIutil(maybe it’s wrong files that I downloaded )and it always show me no command.

 

[Adam Chuan]

As @joevt mentioned, it would be worth while to use pcitools to verify the speed link. Although, based on your screenshots, it looks like you adapter is stuck at an x8 connection.

If you are on a timeline for returning/exchanging the SSD7101-a, I would highly suggest doing so.

Running a quick test this morning with a 2 disk 970Pro raid 0 array, I'm seeing:

• max performance: 5218MB/sec Write and 5645MB/sec Read
• Average performance: 4200MB/s Write and 5475MB/s Read
• Occasional hiccups with APFS: 570MB/s Write and 760MB/s Read*

*occasional slowdowns did not occur with - Mac OS Extended (Journaled)

And if I switch to the Squid Carrier Board(https://amfeltec.com/pci-express-gen-2-carrier-board-4-pcie-m-2-ssd-modules/)
Is it better than 7101 or not?
or I need to check the PCI Link width first?

 

[joevt]

In fact, I don’t know how to use the PCIe tools.

And I don’t know these code where can I write into? terminal? or anywhere?

First, do the following commands in Terminal.app (Terminal.app is found in /Applications/Utilities):

svn checkout https://github.com/pciutils/pciutils/trunk pciutils
cd pciutils
make
sudo su
make install
grep /usr/local/sbin /etc/paths || echo /usr/local/sbin >> /etc/paths
update-pciids
exit

The commands do the following:

1. The first command (svn) downloads the pciutils source code to a folder called pciutils in the current folder which is usually the user folder (/Users/your_user_name or just ~ for short).
2. The second command (cd) changes the current directory to the new pciutils directory that was just created.
3. The third command (make) compiles the pciutils source code (you might need Xcode and Xcode command line utilities)
4. The fourth command (sudo su) switches the current to the super user (root)
5. The fifth command (make install) copies the pciutils files to the various folder (/usr/local/sbin)
6. The sixth command (grep) updates the /etc/paths files to add the /usr/local/sbin directory to the list of paths (if it doesn't exist in the list of paths yet). This is necessary to run the pciutils commands (lspci, setpci, and update-pciids) without entering their full path.
7. The seventh command (update-pciids) updates the pci ids database (which originates from The PCI ID Repository)
8. The eighth command (exit) exits from super user mode

After the above has completed successfully, close the Terminal window and open a new Terminal window. Run the ./pcitree.sh command and copy the text results in a reply to this thread.

I have try so many times to use the code in the PCIutil(maybe it’s wrong files that I downloaded )and it always show me no command.

You should post a screen shot of what happens if you encounter a problem.

And if I switch to the Squid Carrier Board(https://amfeltec.com/pci-express-gen-2-carrier-board-4-pcie-m-2-ssd-modules/)
Is it better than 7101 or not?
or I need to check the PCI Link width first?

They are basically the same cards with slightly different designs. You need to check the PCIe link width and link speed.

 

[w1z]

@Adam Chuan Are you sure you are selecting the Raid0 3-SSD volume in the benchmarking app? Also, can you remove 1 of the SSDs (so the total number of SSDs attached to the card is 3 ... 1 for system and 2 for raid0) and benchmark a Raid0 2-SSD volume?