10GBe speed issues on macOs Sierra with Myricom 10G-PCIE-8C-T

[Daleyelama]

We are running 4 MacPro's 5.1 on a NAS via 10GBe Myricom 10G-PCIE-8C-T network adapter. After upgrading to macOs Sierra there was a maximum speed of 90 MB/s available.

Myricom told us, it's a driver problem. But that's simply not true.
We wrote a script which increases the speed to 1000 MB/s.

If there's anyone out there with the same issues, just send a PN.

 

[pastrychef]

I'm using a SolarFlare 10GbE adaptor with Sierra and have had no problems.

 

[AidenShaw]

I'm using a SolarFlare 10GbE adaptor with Sierra and have had no problems.

I have a vSphere cluster where each ESXi server has six QLogic 57810 10GbE adaptors and all are fine.

(Point - the OP was about Myricom issues - questionable value to say that other cards are OK.)

 

[pastrychef]

OP said it's not drivers. That implies an issue with Sierra. If other 10GbE cards are working without problems, then it may not be Sierra.

 

[Daleyelama]

I'm using a SolarFlare 10GbE adaptor with Sierra and have had no problems.

Which SolarFlare adaptor are you using?

 

[pastrychef]

SFN5122F

 

[Squuiid]

We are running 4 MacPro's 5.1 on a NAS via 10GBe Myricom 10G-PCIE-8C-T network adapter. After upgrading to macOs Sierra there was a maximum speed of 90 MB/s available.

Myricom told us, it's a driver problem. But that's simply not true.
We wrote a script which increases the speed to 1000 MB/s.

If there's anyone out there with the same issues, just send a PN.

AFP or SMB?
SMB does not work for me in Sierra, be it using a SolarFlare SFN5122F or Intel X520-DA2. I've even disabled SMB signing and it's still troublesome. Has been this way since El Capitan for me, and continues in Sierra.

AFP on the other hand gives full speeds over 10GbE.

 

[Daleyelama]

AFP or SMB?

We are running SMB on full speed.

 

[Squuiid]

AFP or SMB?

We are running SMB on full speed.

Wow, that's impressive. I'd love to see the script, could you paste for all to see?

On a separate note, here's my guide on how to use Intel 10GbE NICs in Macs
https://forums.macrumors.com/thread...-nics-to-use-small-tree-os-x-drivers.1968456/

 

[pastrychef]

I have always used AFP.

 

[Daleyelama]

Wow, that's impressive. I'd love to see the script, could you paste for all to see?

On a separate note, here's my guide on how to use Intel 10GbE NICs in Macs
https://forums.macrumors.com/thread...-nics-to-use-small-tree-os-x-drivers.1968456/

Thanks a lot.. VERY INTERESTING!! I don't want to write the script here - even if it's not such a secret.. send you a PN ;-)

 

[Squuiid]

LOL, thanks, but that script you sent me just disables SMB signing. It's all over the internet and it rather goes against the spirit of these forums not to share it.
Regardless, I'd already tried that and while it does improve SMB speeds it's nowhere near as good as AFP.
Thanks anyway.

http://kb.promise.com/thread/disable-smb-signing-on-os-x-10-11-5-10-12-x/

 

[Daleyelama]

LOL, thanks, but that script you sent me just disables SMB signing. It's all over the internet and it rather goes against the spirit of these forums not to share it.
Regardless, I'd already tried that and while it does improve SMB speeds it's nowhere near as good as AFP.
Thanks anyway.

http://kb.promise.com/thread/disable-smb-signing-on-os-x-10-11-5-10-12-x/

And which upload speed do you have with AFP und MacOs Sierra? With SMB we reach 1000MB/s.. with AFP 250MB/s.

 

[pastrychef]

I get about 800MB/s reads and about 300MB/s writes over AFP but that's because I am using extremely slow drives.

 

[Daleyelama]

I get about 800MB/s reads and about 300MB/s writes over AFP but that's because I am using extremely slow drives.

You should try SMB to increase your speed ;-)

 

[pastrychef]

I'm using Seagate 8TB Archive drives. Nothing will help speed those up. lol

I will begin swapping them out for 10TB drives at the beginning of next year which should help a lot.

 

[Daleyelama]

Strange, that you have 800 down and 300 up..! Even if your drives are slow. We use WD8001FFWX. They are great.

 

[Amorphous]

@pastrychef @Daleyelama I wonder how you guys achieve those speeds?

In our Mac Pro 5,1 we have these installed:
- 'Solarflare SFN5122F' which gives 10GBe over optical fibre
- 'HighPoint SSD7101A-1 NVMe M.2 RAID Controller' with EVO 960 NVMe2 sticks, which is the shared volume

However, we are only getting speeds that are really really slow. Our terminals are connected to a Zyxel 10GBe XS1920 switch via CAT7.

Any ideas on how to improve these speeds?

 

[pastrychef]

@pastrychef @Daleyelama I wonder how you guys achieve those speeds?

In our Mac Pro 5,1 we have these installed:
- 'Solarflare SFN5122F' which gives 10GBe over optical fibre
- 'HighPoint SSD7101A-1 NVMe M.2 RAID Controller' with EVO 960 NVMe2 sticks, which is the shared volume

However, we are only getting speeds that are really really slow. Our terminals are connected to a Zyxel 10GBe XS1920 switch via CAT7.

Any ideas on how to improve these speeds?

If you are using the HighPoint RAID Controller in your MacPro5,1, there's a good chance that speeds are being bottlenecked by PCI-e slots. On the MacPro5,1 itself, what kind of speeds are you seeing natively (not over the network)?

The ~300MB/s writes and ~800MB/s reads that I get are from my NAS. The NAS has eight 3.5" SATA hard drives of which two are used for redundancy.

 

[Amorphous]

Hi @pastrychef , thanks for your response.

We have the Highpoint and Solarflare in the PCIe 16x slots, which should theoretically give a bandwidth of 3.2GB/s.

The read/write speed we are achieving locally in the server much faster, (1590MBs Write, 2360MBs Read) screenshot is enclosed below.

The said server has BootROM version 140.0.0.0 with OS10.11.6 El Capitan, and our client terminals run on latest BootROM, uses NVMe to boot OS10.14.

@h9826790, do you have anything you may be able to add here too

 

[pastrychef]

Hi @pastrychef , thanks for your response.

We have the Highpoint and Solarflare in the PCIe 16x slots, which should theoretically give a bandwidth of 3.2GB/s.

The read/write speed we are achieving locally in the server much faster, (1590MBs Write, 2360MBs Read) screenshot is enclosed below.

The said server has BootROM version 140.0.0.0 with OS10.11.6 El Capitan, and our client terminals run on latest BootROM, uses NVMe to boot OS10.14.

@h9826790, do you have anything you may be able to add here too

I can't think of a reason why you are not getting better speeds...

I switched from the SolarFlare SFP+ cards to AQC107 based 10GBase-T cards and I still get pretty much the same speeds...

 

[Squuiid]

Hi @pastrychef , thanks for your response.

We have the Highpoint and Solarflare in the PCIe 16x slots, which should theoretically give a bandwidth of 3.2GB/s.

The read/write speed we are achieving locally in the server much faster, (1590MBs Write, 2360MBs Read) screenshot is enclosed below.

The said server has BootROM version 140.0.0.0 with OS10.11.6 El Capitan, and our client terminals run on latest BootROM, uses NVMe to boot OS10.14.

@h9826790, do you have anything you may be able to add here too

Upgrade your server to 10.14 or at least Sierra and then try again.
Also, I'd do this if under macOS 10.13.3:
https://support.apple.com/en-us/HT205926

Separately, I'd also update that BootROM to 144.0.0.0.0. There are some nice fixes in it.

 

[pastrychef]

Upgrade your server to 10.14 or at least Sierra and then try again.
Also, I'd do this if under macOS 10.13.3:
https://support.apple.com/en-us/HT205926

Separately, I'd also update that BootROM to 144.0.0.0.0. There are some nice fixes in it.

Ah!! Yes!!

I forgot about the potential SMB issues... I've continued to use AFP throughout all these years...

 

[Kier-XF]

One thing I've noticed with 10Gbe is that running any form of virtualization software on my iMac Pro massively impacts ethernet throughput. I've verified this with VMware Fusion Pro, Parallels and Virtualbox - even before they actually spin up virtual machines.

The problem is so enormous that I've actually moved my Vagrant/VMware development suite off my own machine and put it onto a separate box so that my iMP isn't impacted by whatever is going on.

 

[Amorphous]

Thanks guys for all the feedback. At the previous time of last posting, our server was running 10.11 El Capitan.

We hadn't upgraded to newer versions of the OS since we didn't want to depart too far from the 'last supported' version of the 'Solarflare SFN5122F' network card.

The 'Last Supported' version of the card was meant to be 10.9 Mavericks.

Today, we upgraded the server to 10.12 Sierra (for reasons not relevant to this current discussion). And we found the culprit of speed limitation is in fact the Solarflare driver.

After the upgrade, we are getting much slower speeds into the server (see screen shot), we have checked all the internal write speeds and were as per before. The only bottleneck is therefore the solarflare SFP+ card.

Does anyone know other options for dual SPF+ cards for the Mac Pro, with known working drivers for 10.12 Sierra (or above)?