Just a note about getting SFP+ performance on Catalina using an older MacPro with any SFP+ // SFP28 card ... the only thing that actually worked for me (contrary to what everyone says ... that SMB (samba) is the best protocol) was AFP in Peer-to-Peer and transfers to and from my TrueNAS. FWIW, I will always rule out which protocol works best in the future.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Experiences with SFP28?
- Thread starter Kris Kelvin
- Start date
- Sort by reaction score
Just a note about getting SFP+ performance on Catalina using an older MacPro with any SFP+ // SFP28 card ... the only thing that actually worked for me (contrary to what everyone says ... that SMB (samba) is the best protocol) was AFP in Peer-to-Peer and transfers to and from my TrueNAS. FWIW, I will always rule out which protocol works best in the future.
This is simply false. I tested them all thoroughly, got them all to work, and SMB is the clear winner.
@metacollin @Kris Kelvin
I'm hoping you can help me out a bit. If you're short on time, metacollin could you simply upload your EFI? Perhaps, there is something in OpenCore that's allowing it to work. In a 7,1, I'm just not getting the kext/driver to load.
I'm hoping you can help me out a bit. If you're short on time, metacollin could you simply upload your EFI? Perhaps, there is something in OpenCore that's allowing it to work. In a 7,1, I'm just not getting the kext/driver to load.
@metacollin @Kris Kelvin Hope you're well. I'm doing something that is really important and could use your help.Success! I have the TP580-LP-CR running on Ventura 13.2.1!
Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).
No word on speed yet, but it works fine with cxgbtool at least.
I'm afraid I currently don't have the hardware to test any of this.
What you could try is using verbose kextload (kextload -v) and report back with the output.
italy@Italys-Mac-Pro Extensions % kextload -v cxgb.kextI'm afraid I currently don't have the hardware to test any of this.
What you could try is using verbose kextload (kextload -v) and report back with the output.
Executing: /usr/bin/kmutil load -p /Library/Extensions/cxgb.kext
Error Domain=KMErrorDomain Code=71 "Insufficient permissions for action: Unable to load 'com.softraid.driver.SoftRAID' in unprivileged request." UserInfo={NSLocalizedDescription=Insufficient permissions for action: Unable to load 'com.softraid.driver.SoftRAID' in unprivileged request.}
italy@Italys-Mac-Pro Extensions % csrutil status
System Integrity Protection status: disabled.
italy@Italys-Mac-Pro Extensions % sudo kextload -v cxgb.kext
Password:
Executing: /usr/bin/kmutil load -p /Library/Extensions/cxgb.kext
I forgot sudo the first time, but with sudo there is no output and nothing happens.
There is no error shown, so I assume it's loaded.
Can you run the command kextstat to verify that?
If "com.chelsio.driver.cxgb" is listed, it is loaded and we can proceed to the next issue.
PS: Just to be sure, you're on Intel, right? The Chelsio driver currently isn't available for Apple Silicon.
Sonnet introduced a 25G adapter for macOS, too:
https://www.sonnettech.com/product/twin25g/overview.html
Soon to be available as PCIe card or external Thunderbolt adapter (with separate power brick, unfortunately).
It's based on a Mellanox chipset and may currently be the only solution with a modern driver (based on DriverKit).
Apple Silicon and macOS Ventura are supported.
https://www.sonnettech.com/product/twin25g/overview.html
Soon to be available as PCIe card or external Thunderbolt adapter (with separate power brick, unfortunately).
It's based on a Mellanox chipset and may currently be the only solution with a modern driver (based on DriverKit).
Apple Silicon and macOS Ventura are supported.
Thank you for helping me out. Intel, yes. It's an Intel Xeon W processor in a Mac Pro 7,1 2019.There is no error shown, so I assume it's loaded.
Can you run the command kextstat to verify that?
If "com.chelsio.driver.cxgb" is listed, it is loaded and we can proceed to the next issue.
PS: Just to be sure, you're on Intel, right? The Chelsio driver currently isn't available for Apple Silicon.
I don't see it, but here are two screenshots.
I also have it working with the official driver from Chelsio on Big Sur (they list it for Catalina, but I managed to get it working in Big Sur, but not in Monterey or Ventura) if that's helpful.
Attachments
Last edited:
Good to know, but they do cost $900, and only 25G. I like to keep 40G at my minimum these days.
@metacollin - not sure if you could upload your Supermicro EFI. What model motherboard is it?There is no error shown, so I assume it's loaded.
Can you run the command kextstat to verify that?
If "com.chelsio.driver.cxgb" is listed, it is loaded and we can proceed to the next issue.
PS: Just to be sure, you're on Intel, right? The Chelsio driver currently isn't available for Apple Silicon.
@Kris Kelvin Any thoughts to proceed?
The kext doesn't seem to be loaded. You'll have to find out why. Maybe because it is (invalidly) signed, maybe it is in the wrong location, maybe SIP isn't really off. There are many possibilities, but this isn't really the thread for that. Google is your friend.
It is a Mellanox CX4. I didn't modify anything. I settled with just using my X520-DA2 since it's Plug n Play with 10G working.
That's a bummer, I was hoping you had more to offer than that, but I understand you may be busy. SIP is definitely turned off as I showed you the screenshot, the kext is definitely in the correct place, so that leaves us with signing it. I think someone mentioned needing a Developer account to sign it, SO if anyone is reading this... does anyone have a Developer account with Apple?The kext doesn't seem to be loaded. You'll have to find out why. Maybe because it is (invalidly) signed, maybe it is in the wrong location, maybe SIP isn't really off. There are many possibilities, but this isn't really the thread for that. Google is your friend.
@metacollin Still hoping to hear from you as your EFI or advice may solve this.
Some people figured out that macOS Ventura already contains DriverKit-based drivers for additional ethernet chipsets out of the box: Mellanox (ConnectX-4, ConnectX-5 and ConnectX-6) and Intel (X500 and X700).
This post here says it works fine (except for not supporting jumbo frames), even in a Thunderbolt PCIe enclosure.
This is great news and now clearly the best option going forward. No tinkering with SIP or third-party drivers and no risky system upgrades!
I'm off to find some used Mellanox gear.
This post here says it works fine (except for not supporting jumbo frames), even in a Thunderbolt PCIe enclosure.
This is great news and now clearly the best option going forward. No tinkering with SIP or third-party drivers and no risky system upgrades!
I'm off to find some used Mellanox gear.
Edit: nevermind, the issue was on my side with the switch. Works great, thanks again!Success! I have the TP580-LP-CR running on Ventura 13.2.1!
Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).
No word on speed yet, but it works fine with cxgbtool at least.
Hey Colin, nice work! I used your kext and was able to also get Ventura to (at least) "see" my TP580-LP-CR card too. However, were you able to get it to actually talk on the network? Mine while showing a link up in ifconfig, doesn't actually connect to the network - can't DHCP - and I think it is because the link speed is not correct, but I cannot change it in the "hardware" section of network utils. Did you make any progress here? Feels like we are so close. I am using intel hackintosh with OC bootloader.
Last edited:
A $300 small external Thunderbolt 25G Ethernet Adapter for Macs:
Enclosure: Wikingoo M1 (~$200 from Aliexpress, search for "Mini egpu")
Adapter: Nvidia Mellanox ConnectX-4 Lx (MCX4121A-ACAT with low-profile bracket) (~$100 from Ebay)
Driver: Apple's Mellanox driver (integrated in macOS Ventura)
Some notes:
Enclosure: Wikingoo M1 (~$200 from Aliexpress, search for "Mini egpu")
Adapter: Nvidia Mellanox ConnectX-4 Lx (MCX4121A-ACAT with low-profile bracket) (~$100 from Ebay)
Driver: Apple's Mellanox driver (integrated in macOS Ventura)
Some notes:
- The ethernet adapter came with a Huawei firmware; I flashed the regular Mellanox firmware to it (using a Linux box).
- The ConnectX-4 Lx consumes around 10 watts, so it needs some airflow (see manufacturer specifications). The Wikingoo M1 doesn't have any active cooling, so I had to find way to install a fan. Luckily, the board contains an EPS12V connector (alternative power input?). It can be utilized as a 12V power source for fans. I've installed two 60x60x10mm fans that ensure both the ethernet chip and the pluggables will stay cool (at the expense of some noise, despite using Noctua NA-SRC10 low noise adapters). Maybe the QNAP QXG-25G2SF-CX6 PCIe card would be a better fit – same Mellanox chipset but with an integrated fan.
- I'd assume that performance is not quite as good as the (pretty expensive) solutions from ATTO and Sonnet. However, I like the ability to use the device without any additional driver.
- The Thunderbolt bridge is bus-powered, but the PCIe card is powered by an external 12V power supply. The Thunderbolt spec only guarantees 15 watts of power anyway, so powering a ConnectX-4 over a Thunderbolt connection is probably impossible.
Last edited:
Success! I have the TP580-LP-CR running on Ventura 13.2.1!
Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).
No word on speed yet, but it works fine with cxgbtool at least.
Oh nice, I'm not the only one who looked into this 🙂 I ended up doing the same thing in a Lilu plugin: https://github.com/al3xtjames/CXGBFixup
I also saw that the link would drop after a sleep/wake cycle when using jumbo frames. I ended up hacking around this by reconfiguring the MTU on wake (by hooking functions with Lilu).
Hey thanks for posting this, it was really helpful to know that an mlx5 driver is in macOS now and it allowed me to pull the trigger on a ConnectX-4 card. I was able to order a MCX455A-ECAT for around $70, this is actually a 100Gbit card with the 16 PCIe lanes as well, I felt good about one of the cheapest available options being a 100Gbit capable card, so I don't have to even flash it to run that speed. Also, it is known that almost all of these CX4 cards can be flashed to unlock 100Gbit on them!
Still, the dual port cards are all at around the $140 mark or higher so I am grateful to get something functional for my Apple Silicon macbook without breaking the bank, since this time all I needed to buy was the CX4 card, which is perfect. It works with this old Sonnet eGPU enclosure I have. it's comically big for this tiny card, and the card I have does not have a full height bracket, but it works! It did not show any signs of life until I set it to Ethernet mode with `mlxconfig` with my linux machine. But I am sure this workflow only affects VPI cards that are capable of Infiniband, they tend to come configured for IB by default.
I haven't configured packet size and I am getting 12Gbit between this CX4 and a CX3 on my linux box. I'm not sure if that's to be expected or not.
What configuration did you do to push nearly 20Gbit with yours? I'm worried that jumbo packet size configuration is the only knob we have to tweak. It does not seem like RDMA will come to macOS any time soon. Not sure.
At any rate even with roughly 10Mbit it's already worth drilling some holes and running fiber into my living room, so I can finally have a handy USB-C cable that charges the macbook while also giving it full gigabit internet speed as well as let me actually transfer data faster than I can with external storage.
Default configuration, actually. No Jumbo frames or anything.
Note that the 19Gbit/s were from "remote" (PC) to "local" (Mac). The other way may have been slower.
What was the negotiated Ethernet link speed between the ConnectX-3 and the ConnectX-4?
Ah ok. I didn't actually test MTU 1500, in reverse, but I'll test next time, yes I had 9000 MTU set on the Linux connectx-3 side, and with iperf3 -R (to send from linux) I do get 19.9Gbps so that lines up with your result.
With MTU 2034 (the max allowed by Apple's driver) I can only send at 16Gbit/s. It's not bad, but a bit frustrating knowing the hardware can do more.
Negotiated speed is 40Gbit.
I suppose I'll just have to settle for this. As I can only really afford 4 PCIe lanes to put these NICs on... even if there wasn't this bottleneck, there'd be the thunderbolt inherent bottleneck which I seem to recall is around 22Gbps, and then, shortly after that comes the 4 PCIe lane bottleneck which I measured at 26Gbit between linux and windows back when I was testing that. Where both ends were connected on M.2 slots with 4 lanes.
With MTU 2034 (the max allowed by Apple's driver) I can only send at 16Gbit/s. It's not bad, but a bit frustrating knowing the hardware can do more.
Negotiated speed is 40Gbit.
I suppose I'll just have to settle for this. As I can only really afford 4 PCIe lanes to put these NICs on... even if there wasn't this bottleneck, there'd be the thunderbolt inherent bottleneck which I seem to recall is around 22Gbps, and then, shortly after that comes the 4 PCIe lane bottleneck which I measured at 26Gbit between linux and windows back when I was testing that. Where both ends were connected on M.2 slots with 4 lanes.
I used your kext, and was able to get it working (ish) in Sonoma, but couldnt get it in Ventura for whatever reason. Card shows as "Terminator 4" lol. Card is identified but can get the link upSuccess! I have the TP580-LP-CR running on Ventura 13.2.1!
Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).
No word on speed yet, but it works fine with cxgbtool at least.
I just tested two 25GB sfp28 nics through Sonnet echo express single slot low profile expansion TB3 enclosure.
Frist card was Intel XXV710-DA1. I do not recommend using it. It did not properly work with my mikrotic CRS510-8XS-2XQ-IN switch. I could access the internet and some hosts on my network, but it did something, that I could not even ping my ESXi server and all VMs running on it. What is even more bizarre is that other computers on the network could not ping ESXi host and VMs on it when this card was active on the network. I tried all possible configurations for the SFP28 port on the switch, but nothing helped.
Second card I tried was Mellanox MCX4111A-ACAT, also single port 25GB sfp28. This card had no issues what so ever. I could access all my devices on the network. So I am sticking with it, just need to find a low profile bracket for it.
Intel card used native macOS driver com.apple.DriverKit-AppleEthernetXL while Mellanox used com.appleDriverKit-AppleEthernetMLX5.
Frist card was Intel XXV710-DA1. I do not recommend using it. It did not properly work with my mikrotic CRS510-8XS-2XQ-IN switch. I could access the internet and some hosts on my network, but it did something, that I could not even ping my ESXi server and all VMs running on it. What is even more bizarre is that other computers on the network could not ping ESXi host and VMs on it when this card was active on the network. I tried all possible configurations for the SFP28 port on the switch, but nothing helped.
Second card I tried was Mellanox MCX4111A-ACAT, also single port 25GB sfp28. This card had no issues what so ever. I could access all my devices on the network. So I am sticking with it, just need to find a low profile bracket for it.
Intel card used native macOS driver com.apple.DriverKit-AppleEthernetXL while Mellanox used com.appleDriverKit-AppleEthernetMLX5.
Would love to get some help on my setup.
2021 M1 Max MBP
Sonoma 14.6
Thunderbolt 3/4 to PCI enclosure:
1. Sonnet eGFX breakaway box (I think this has a JHL-8440 chip)
2. Trebleet mini eGPU board (JHL-7440 chip)
NIC:
1. Mellanox ConnectX-5 MCX555A-ECAT connected via 100Gbase DAC to ConnectX-5 on Ubuntu server where it is getting all 16 lanes.
Max MTU allowable is 2034
iperf3 tests are consistently 8.95 Gbits/sec no matter what I do.
Here is what I tried:
1. Two different enclosures
2. Various cables
3. -P8 iperf3 test, nutttcp test, etc.
4. 4 different cables
Any help would be appreciated.
2021 M1 Max MBP
Sonoma 14.6
Thunderbolt 3/4 to PCI enclosure:
1. Sonnet eGFX breakaway box (I think this has a JHL-8440 chip)
2. Trebleet mini eGPU board (JHL-7440 chip)
NIC:
1. Mellanox ConnectX-5 MCX555A-ECAT connected via 100Gbase DAC to ConnectX-5 on Ubuntu server where it is getting all 16 lanes.
Max MTU allowable is 2034
iperf3 tests are consistently 8.95 Gbits/sec no matter what I do.
Here is what I tried:
1. Two different enclosures
2. Various cables
3. -P8 iperf3 test, nutttcp test, etc.
4. 4 different cables
Any help would be appreciated.
- What's the speed of the Thunderbolt link (as shown in System Profiler)?
- What's the speed of the link (as shown in System Settings)?
- Measured 8.95 Gbits/sec in both directions?
- Is a single core of the CPU on either side maxed out when testing?