MP 1,1-5,1 Modify retail Intel 10GbE NICs to use Small Tree macOS Drivers

[Squuiid]

FOLKS, THIS WILL WRITE TO THE CARD EEPROM AND INCORRECT USAGE COULD RENDER YOUR CARD USELESS. THESE STEPS ARE SPECIFIC TO MY INTEL X520-DA2 10GbE CARD ONLY. HOWEVER, WITH SOME WORK THEY CAN BE ADAPTED TO WORK ON MOST INTEL NETWORK CARDS, BOTH 1GbE and 10GbE.
PROCEED AT YOUR OWN RISK.

Intel's own network driver page states:
Intel adapters from Small Tree Communications are uniquely designed and validated on Apple platforms with OS X* drivers. Software and drivers for Mac OS X* for retail versions of Intel® Ethernet Adapters are not available from Intel or from Small Tree Communications.

In order for a retail Intel NIC to be able to use the Small Tree macOS drivers it must have a Subsystem ID of 000a, as stated here:
https://www.small-tree.com/support/...now-which-Gigabit-driver-to-install-on-my-Mac

Fortunately, a card's Subsytem ID can be modified by using the 'ethtool' command under Linux.
The following is how I modified my card.

First step, download and install Small Tree macOS NIC drivers.
https://small-tree.com/support/downloads/

Next, download and boot off of Ubuntu 16.04 LTS Desktop CD and run terminal.
https://ubuntu.com/download/alternative-downloads
Don’t use a newer version of Ubuntu as the newer version of ethtool does not work.

TO LOCATE ethX:
ifconfig

TO BACKUP EEPROM (DO THIS!):
sudo ethtool -e ethX raw on > ethX.bin

COMMAND TEMPLATE WE NEED TO USE TO CHANGE OFFSET VALUES:
sudo ethtool -E ethX magic 0x<device id><vendor id> offset 0x<offset> value 0x<value>

TO DETERMINE VENDOR, DEVICE AND SUBSYSTEM IDs:
lspci -nn -vvv | grep Ethernet
01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:7a11]
01:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:7a11]

NOW, TO FIND THE RELEVANT OFFSETS!
Intel datasheet for my card states the following, but it doesn't seem to be correct! Other datasheets may prove more useful, it's definitely worth checking them first. Look under the 'EEPROM Hardware' section of the datasheet:
PCIe Sub-System ID - Offset 0x08
Bits:15:0
Name:Sub System ID
Default: 0x0

By running the following command and eyeballing the offsets I've discovered that the values below are actually the two relevant values on my Intel X520-DA2 card. This is the subsystem ID we're after (notice the vendor id of 86 80 which follows it):
sudo ethtool -e eth1 | less
"0x0320: 03 00 1f 00 00 00 00 2b 03 13 11 7a 86 80 a6 10"

So, if we start at offset 0x0320 you would count 0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f for each pair of digits.
The values we need to change are 11 at 0x032a and 7a at 0x032b. We can remove the 0 to shorten to 0x32a and 0x32b

So, for an Intel X520-DA2 (8086:10fb)(8086:7a11), to set subsys ID of 0x000a we would run:
sudo ethtool -E eth1 magic 0x10fb8086 offset 0x32a value 0x0a
sudo ethtool -E eth1 magic 0x10fb8086 offset 0x32b value 0x00

sudo ethtool -E eth2 magic 0x10fb8086 offset 0x32a value 0x0a
sudo ethtool -E eth2 magic 0x10fb8086 offset 0x32b value 0x00

In order to allow ethtool to write to the EEPROM you need the 'magic' value specific to your card, which is:
0x<device id><vendor id>
The command below will show these values:
lspci -nn -vvv | grep Ethernet
NOTE these are not the Subsystem values, they are the Device and Vendor IDs, in my case 10fb (X520) and 8086 (Intel).
Here's a useful reference:
http://pci-ids.ucw.cz/read/PC/8086

Reboot and done!

UPDATE:
Make sure you disable SMB signing if you require full performance when connecting to a Windows SMB share.
https://support.apple.com/en-us/HT205926

I've now tested on an Intel X540-T1 and it also works fine.

Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X540-T1 [8086:0002]

sudo ethtool -E ens4 magic 0x15288086 offset 0x48e value 0x0a


One more thing. I was using Belkin CAT6 UTP 1m ethernet cables and they weren't good enough for 10GbE. I've since changed them for CAT7 and can now max out 10GbE with ease. Check your cables, 10GbE is not forgiving.

 

[prvt.donut]

That is great news. I was wondering if we could use the x520-da2 card on Mac Pro's. Have you tried connecting to an iscsi storage with it?

 

[Squuiid]

Have you tried connecting to an iscsi storage with it?

No, just direct attach to a single VMware server, which is running Avago 9271-8i RAID cards.
No real need for iSCSI here. Just enjoying 10GbE speeds to my file server
Good luck with it, let us know how it goes.

 

[Squuiid]

UPDATE:
You will likely want to disable the Intel card's Boot ROM. I was unable to use BootCamp until I disabled it.

In order to do this you will need to download Intel's BOOTUTIL and run the following command on a Windows PC:
https://downloadcenter.intel.com/do...s-Boot-Utility-Preboot-Images-and-EFI-Drivers

bootutil -FLASHDISABLE -nic=1
bootutil -FLASHDISABLE -nic=2

If that doesn't work try this first and then retry above:

bootutil -nic=1 -BOOTENABLE=DISABLED
bootutil -nic=2 -BOOTENABLE=DISABLED

See here for BOOTUTIL command switches:
https://www-ssl.intel.com/content/www/us/en/support/software/manageability-products/000005790.html

While you're here, and before disabling the flash using the command above, you could update the firmware of the card.

bootutil -IV -nic=1
bootutil -IV -nic=2
This will show you what firmware is currently installed.

bootutil -UP=COMBO -nic=1
bootutil -UP=COMBO -nic=2
This will update the firmware of your Intel card. It assumes that the BootIMG.FLB firmware file is located in the same folder as the bootutil command.

NOTE: The switches on the bootutil command are CASE SENSITIVE.

 

[fhturner]

To paraphrase Tommy Boy... This is.....AWEsome!!! Based solely on reading your fantastic how-to, Squuiid, I decided to pull the trigger on a pair of X520-DA2s on eBay that included a pair of Direct Attach Copper cables w/ each card for a mere $90. Just got them "flashed" and did some file transfer tests using RAM disks on a couple of Mac Pros and hit over 900MB/s w/ essentially no tweaks, just using personal file sharing. Holy schnikes!! Same functionality new would've cost over $1400. Thanks, Squuiid!

Fred

 

[Squuiid]

Has anyone tried this with other Intel cards, specifically 10GBASE-T X540 based adapters?

 

[Squuiid]

OK, a note about using 10GbE NICs in 10.11.5+ and macOS Sierra.
Apple has enabled SMB signing from 10.11.5 and this has a huge impact on 10GbE network transfers. I'm seeing performance more in line with 2GbE rather than 10GbE when accessing SMB shares.
In order to 'fix' this you need to disable SMB signing on both your NAS/server and macOS.

Windows Servers:
https://redmondmag.com/articles/2014/05/16/network-performance-problems.aspx
macOS:
https://support.apple.com/en-us/HT205926

Doing the above will improve performance of 10GbE in macOS over SMB shares. However, do bear in mind that you are disabling a security feature and consequently opening yourself up to man in the middle attacks.

UPDATE:
macOS Mojave or newer already has SMB signing disabled by default.

 

[fhturner]

OK, a note about using 10GbE NICs in 10.11.5+ and macOS Sierra.
Apple has enabled SMB signing from 10.11.5 and this has a huge impact on 10GbE network transfers, essentially rendering them unusable.
In order to 'fix' this you need to disable SMB signing on both your NAS/server and macOS.

Windows Servers:
https://redmondmag.com/articles/2014/05/16/network-performance-problems.aspx
macOS:
http://forum.promise.com/thread/fix-for-smb-sanlink-2/

Doing the above will fully restore functionality of 10GbE in macOS. However, do bear in mind that you are disabling a security feature and consequently opening yourself up to man in the middle attacks.

I saw this on a Mac Pro I set up as a new server for a client of mine recently. Have been meaning to post results about it, so I'll list a few details here... I noticed that testing using RAM disks with 10.10 and 10.11 non-server hosts and clients was going very fast using SMB (~800MB/s+) and even faster via AFP (>1GB/s). But when I configured the server machine w/ the actual Server.app, the SMB transfers (which the OS defaults to when you click on the server in the Finder sidebar) dropped to 200MB/s— better than GigE's ~120MB/s, but not even by a factor of 2x. The AFP transfers stayed very fast.

So I troubleshot and experimented before arriving at the same discovery— that 10.11 Server was turning on SMB signing. Disabling this returned SMB transfers to the near-1GB/s realm I had previously seen. However, I did further testing.

Turns out this 200MB/s "cap" was due to how fast a single thread could go on the CPU. Squuiid, you may have seen less than 200MB/s if you were on a slower CPU; this system was a Hex 3.46GHz (upgraded Quad 2.66GHz), so about as fast as a single thread could go. But I found that if I added clients, each of them could increase the total throughput w/ add'l thread usage. I only had one other 10GbE to test with, but I added 2 more 1Gbps clients to the mix as well. With a 200MB/s transfer already going, the add'l clients could run the total throughput up to 400MB/s+ by adding their own 120MB/s transfers. Additionally, if I ran a 1GB/s AFP transfer w/ the 10GbE client, I could also get another 200MB/s or so by doing transfers from the other 2 1Gbps clients at the same time.

Fred

 

[Squuiid]

Turns out this 200MB/s "cap" was due to how fast a single thread could go on the CPU. Squuiid, you may have seen less than 200MB/s if you were on a slower CPU; this system was a Hex 3.46GHz (upgraded Quad 2.66GHz), so about as fast as a single thread could go.

Are you saying that signing each packet maxes out a single core of the CPU, whereas unsigned packets don't tax the CPU as much?
Windows has Receive Side Scaling which forces utilisation of multiple CPU cores for network transfers. Anything comparable on a Mac?
I'd be surprised if SMB signing maxes out a single 3.46GHz CPU core when lesser devices such as NAS units have no problem with this.

 

[fhturner]

Are you saying that signing each packet maxed out a single core of the CPU, whereas unsigned packets don't tax the CPU as much?
Windows has Receive Side Scaling which forces utilisation of multiple CPU cores for network transfers. Anything comparable on a Mac?
I'd be surprised if SMB signing maxes out a single 3.46GHz CPU core when lesser devices such as NAS units have no problem with this.

I could be mistaken, but as far as I could tell, it was CPU-bound. I believe it was pegging a kernel task or other process (it's been a few weeks, so I'm fuzzy on which) @ 100% when signed, and less when unsigned. Maybe the signing is poorly executed in Apple's SMB code?

When tackling this, I had read about the RSS and multiple-core SMB transfers as well. I was surprised to find that AFP outperformed SMB (w/o signing) by 200MB/s or so in my testing, even though the talk was that AFP was dead-end and not going to be taking advantage of multiple cores. Turns out it didn't really need it. Granted, I was looking for peak speeds, so I was using large files; with small files, maybe SMB overtakes AFP, but I don't think I would've seen 1GB/s+. I was only using Macs for this testing, so I can't say how a Windows client (or server) might have done w/ my equipment.

On those NAS units, are you sure you're talking about signed SMB transfers? I'm not that familiar w/ signing or how compute-intensive that would be. I seem to recall the number of packets sent & received/sec when signing being significantly higher than AFP or unsigned SMB.

Fred

 

[Squuiid]

Maybe the signing is poorly executed in Apple's SMB code?

Yep, I very much think this is the issue here and given the lack of 10GbE cards in Apple's current Mac range I doubt they'll be rushing to fix this unfortunately.

 

[fhturner]

Yep, I very much think this is the issue here and given the lack of 10GbE cards in Apple's current Mac range I doubt they'll be rushing to fix this unfortunately.

Well, with eleventy billion $$ in the bank, their cash reserves could only support a few thousand more programmers; and what do you think those programmers would be focused on anyway?? There are especially no 10GbE ports on iPhones!

 

[pablos]

This is such a great job you did Squuiid!!! I've been looking for some budget solution to build a 10GBE/40GBE network, and now I think I will try it with aggregating ports in dual intel x520-DAs. Do you think this hack would work on Intel-X710DA4?

 

[PowerMac84]

Tested with a Dell branded X520 T2 (Base T-Ethernet) und a China no-name X540 T2 (130 € from Ebay, passively cooled).

I‘d like to add that lspci may need | grep Network instead of | grep Ethernet. For some cards, some commands have to be added to lspci to find out the vendor ID and subsystem ID, e.g. -D or -i.

Sure, one has to find the fitting offsets, for example, it was x480 on the X540 card.

 

[Squuiid]

I've now tested on an Intel X540-T1 and it also works fine.

Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X540-T1 [8086:0002]

sudo ethtool -E ens4 magic 0x15288086 offset 0x48e value 0x0a

Done!

 

[cclarke99]

Success with Intel x520-DA1

@Squuiid -- thank you for these awesome instructions! Last night, I successfully rewrote the values on an x520-DA1 card. I ran this with the latest Ubuntu Mate live image, so my interface name is different but this is nearly identical to the x520-DA2 mentioned in the OP

Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-1 [8086:0006]

sudo ethtool -E enp4s0 magic 0x10fb8086 offset 0x32a value 0x0a
sudo ethtool -E enp4s0 magic 0x10fb8086 offset 0x32b value 0x00

 

[yurc]

Hello @Squuiid, or anyone can help me to check is my command correct?

My card is not 10GbE, just plain 1GbE Intel NIC (I350-T4). I don't have many experience in Linux command tools, so I'm need some expert to make sure if my procedure is correct. I'm also attach my offset result in text format.

I350-T4 :
Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter I350-T4 [8086:0001]

Offset:

I'm still not flashed it. So is this value correct for I350-T4?

sudo ethtool -E enp1s0f0 magic 0x15218086 offset 0x0016 value 0x0a

 

[Squuiid]

Hello @Squuiid, or anyone can help me to check is my command correct?

My card is not 10GbE, just plain 1GbE Intel NIC (I350-T4). I don't have many experience in Linux command tools, so I'm need some expert to make sure if my procedure is correct. I'm also attach my offset result in text format.

I350-T4 :
Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter I350-T4 [8086:0001]

Offset:

View attachment 697526

I'm still not flashed it. So is this value correct for I350-T4?

sudo ethtool -E enp1s0f0 magic 0x15218086 offset 0x0016 value 0x0a

Yep, that looks good to me.
Curious as to what you need more than the 2 inbuilt 1GbE NICs for. VMware? Or is this for a Hackintosh?

 

[yurc]

Yep, that looks good to me.
Curious as to what you need more than the 2 inbuilt 1GbE NICs for. VMware? Or is this for a Hackintosh?

Yes, it worked!! Not tested in OSX though but checking lspci from linux gave me 0x000a subsytem ID now, so i think it will work flawlessly once I'm plug in OSX and install Small Tree kext.

Yes, your last guess is right. Primarily for hackintosh, but since this hack are permanent on hardware, i can utilize it for real mac pro too and more convenient rather than injector kext/dummy kext method/unsigned kext or such.

My hackintosh is build using cheesegrater case, but I'm retain original rear port (no cutting) so access to rear connectivity it's pretty limited, so my better choice is only PCIe ports as connectivity. Here some old photos to describe my condition why I'm need discrete NIC

Even I can make rear ports work using extension self-made logic board inside, built-in NIC on my motherboard require me to using unsigned kext which i don't want. With this solution, driver from Small Tree are far better and reliable.

And off course.....thank you for your assistance and this guide!

 

[alexanderdi]

I've now tested on an Intel X540-T1 and it also works fine.

Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X540-T1 [8086:0002]

sudo ethtool -E ens4 magic 0x15288086 offset 0x48e value 0x0a

Done!

FYI, your location worked perfectly with my X540-AT2 despite different subsystem values. The TLDR for anyone who finds this:

------
ifconfig

(should have two very similar entries with es-whatever or the like; each one represents a port location)

sudo ethtool -E LOCATION1 magic 0x15288086 offset 0x48e value 0x0a
sudo ethtool -E LOCATION1 magic 0x15288086 offset 0x48f value 0x00
sudo ethtool -E LOCATION2 magic 0x15288086 offset 0x48e value 0x0a
sudo ethtool -E LOCATION2 magic 0x15288086 offset 0x48f value 0x00
------

Much appreciated!

 

[mijail]

Windows has Receive Side Scaling which forces utilisation of multiple CPU cores for network transfers. Anything comparable on a Mac?

In case anyone cares, I just wanted to correct a misunderstanding I think I see in this thread: Receive Side Scaling is actually not to make various CPUs work together on network transfers. Rather to the contrary: RSS makes sure that traffic for one given connection stays in one given CPU, to maximize usage of cache / minimize disturbing multiple CPUs.

Of course, different connections (probably) end up in different CPUs; but the point is that once one CPU starts working in one connection, the rest of the system tries not to disturb that relationship.

So I would be surprised if RSS helped in a few-users, big-transfer SMB scenario.
[doublepost=1501101598][/doublepost]

Maybe the signing is poorly executed in Apple's SMB code?

Given that Windows has the same problem (and that SMB is Microsoft's protocol), it rather smells like there's some design problem with SMB signing itself ("problem" in that it seemingly doesn't allow for fast-enough implementations in current hardware)

 

[justin lovell]

FYI, your location worked perfectly with my X540-AT2 despite different subsystem values. The TLDR for anyone who finds this:

------
ifconfig

(should have two very similar entries with es-whatever or the like; each one represents a port location)

sudo ethtool -E LOCATION1 magic 0x15288086 offset 0x48e value 0x0a
sudo ethtool -E LOCATION1 magic 0x15288086 offset 0x48f value 0x00
sudo ethtool -E LOCATION2 magic 0x15288086 offset 0x48e value 0x0a
sudo ethtool -E LOCATION2 magic 0x15288086 offset 0x48f value 0x00
------

Much appreciated!

Can someone assist me in getting my XX to work?
I am all ready to go, but can't quite figure out the proper steps to get it setup. We could do a screen share.

I have the x540 - at2 also.

You can pm me or reply to this thread.

I got to this step:
TO LOCATE ethX:
ifconfig
(typed that in) . Then didn't know what to do from here... I typed in the below information - but I think i need to do something / modify this somehow - sorry my mac coding is limited - and not quite sure how to follow this.

TO BACKUP EEPROM (DO THIS!):
sudo ethtool -e ethX raw on > ethX.bin

Thanks!

 

[Squuiid]

Can someone assist me in getting my XX to work?
I am all ready to go, but can't quite figure out the proper steps to get it setup. We could do a screen share.

I have the x540 - at2 also.

You can pm me or reply to this thread.

I got to this step:
TO LOCATE ethX:
ifconfig
(typed that in) . Then didn't know what to do from here... I typed in the below information - but I think i need to do something / modify this somehow - sorry my mac coding is limited - and not quite sure how to follow this.

TO BACKUP EEPROM (DO THIS!):
sudo ethtool -e ethX raw on > ethX.bin

Thanks!

Post what you get from running the commands and we can then assist.

 

[justin lovell]

Post what you get from running the commands and we can then assist.

Hmm..

Tried holding down option when restarting with the usb key in, but nothing happens. I just get a black screen.

Thats the first hurdle..

 

[Squuiid]

Hmm..

Tried holding down option when restarting with the usb key in, but nothing happens. I just get a black screen.

Thats the first hurdle..

It's best to do the above guide on a PC. The BIOS in the card seems to be active on your card and is stopping your Mac from being able to select the boot disk by using the option key.
Find a PC desktop, install your card and then follow the guide.