Experiences with SFP28?

[Kris Kelvin]

Does anyone have any experience with 25GbE (SFP28) on Macs?

The only adapter with official macOS support seems to be ATTO's ThunderLink® NS 3252 (over Thunderbolt 3). I couldn't find any benchmarks, but given Thunderbolt's limitations, I assume it will max out at about 20Gbit/s. I'd love to hear from people using it.

 

[joevt]

Does anyone have any experience with 25GbE (SFP28) on Macs?

The only adapter with official macOS support seems to be ATTO's ThunderLink® NS 3252 (over Thunderbolt 3). I couldn't find any benchmarks, but given Thunderbolt's limitations, I assume it will max out at about 20Gbit/s. I'd love to hear from people using it.

20 Gbps = 2500 MB/s.

I've seen Thunderbolt 3 get 2800 MB/s (22.5 Gbps) from NVMe.

I found this https://docs.broadcom.com/doc/broadcom-ethernet-adapters-deliver-lower-tail-latency
which shows Broadcom and Mellanox chips getting less than 22 Gbps. I suppose Ethernet overhead (and FEC overhead?) might drop the bandwidth below the Thunderbolt 3 max of 22.5 Gbps? What chip does the ATTO ThunderLink NS 3252 use? Inspecting the macOS kext might give a clue (PCI IDs matching).

 

[AdditionalPylons]

I've been wanting to find out more about 25GbE options for Mac as well. I downloaded the driver for ThunderLink NS3252 and used The Unarchiver to extract some of the files from the ATTOThunderLinkNC3_1081f1.pkg.
Inside the ATTOThunderLinkNC3_1081f1/ATTOThunderLinkNC3.pkg/Payload file there are multiple mentions of Mellanox:

L186879: !pbase/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_fs_tree.c!devmlx5_core: %s:%s:%dpid %d): failed to allocate inbox
L186890:
fdb_rootfdb_prioesw_egress_rootesw_egress_prioesw_ingress_rootesw_ingress_priosniffer_tx_rootsniffer_priosniffer_rx_rootmlx5_cmd_fs_set_fte"queue_work on inactive work queue\n"@/Volumes/Development/mellanox/source/cna3/osx/kext/compat/osdep.cpp:174mlx5_core: %s:%s:%dpid %d): MLX5_CMD_OP_MODIFY_VPORT_STATE failed
L187086:
mlx5_get_rsccreate_qprqsq_commonpxd_ready_bppci_write_bppci_read_bppci_read_stuck_no_completion_buffermax_pci_bwused_pci_bwrx_pci_errorstx_pci_errorstx_pci_correctable_errorstx_pci_non_fatal_errorstx_pci_fatal_errorsl0_mtt_missl0_mtt_hitl1_mtt_missl1_mtt_hitl0_mpt_missl0_mpt_hitl1_mpt_missl1_mpt_hitrxb_no_slow_path_creditsrxb_no_fast_path_creditsrxb_rxt_no_slow_path_cred_perf_countrxb_rxt_no_fast_path_cred_perf_countrxt_ctrl_perf_slice_load_slowrxt_ctrl_perf_slice_load_fastrxt_steering_perf_count_steering0_rse_work_raterxt_steering_perf_count_steering1_rse_work_rateperf_count_tpt_creditperf_wb_missperf_wb_hitrxw_perf_rx_l1_slow_miss_ldbrxw_perf_rx_l1_slow_hit_ldbrxw_perf_rx_l1_fast_miss_ldbrxw_perf_rx_l1_fast_hit_ldbrxw_perf_l2_cache_read_miss_ldbrxw_perf_l2_cache_read_hit_ldbrxw_perf_rx_l1_slow_miss_reqslrxw_perf_rx_l1_slow_hit_reqslrxw_perf_rx_l1_fast_miss_reqslrxw_perf_rx_l1_fast_hit_reqslrxw_perf_l2_cache_read_miss_reqslrxw_perf_l2_cache_read_hit_reqslrxs_no_pxt_creditsrxc_eq_all_slices_busyrxc_cq_all_slices_busyrxc_msix_all_slices_busysxw_qp_done_due_to_vl_limitedsxw_qp_done_due_to_deschedsxw_qp_done_due_to_work_donesxw_qp_done_due_to_limitedsxw_qp_done_due_to_e2e_creditssxw_packet_send_sxw2sxp_go_vldsxw_perf_count_steering_hitsxw_perf_count_steering_misssxw_perf_count_steering_rse_0sxd_no_sched_creditssxd_no_slow_path_sched_creditstpt_indirect_mem_keymlx5_alloc_db_from_pgdir(pgdir, db)/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_alloc.cHardwareLroHardware LROEnable hardware LRO.
L187137:
œÌÒ
*ÒLÒmÒÄÒóÒÆÒ«Ò ÂÒ Ú Ú 8Ú WÚ vÚ íÚ ØÚ ÕÚ$‚Ú$Û(v¥–.|¥ÿ.Ç¥‡.à¥Ë.é¥..ö¥/†¥/¶¥/k¥p «¥/Ω¥"¥ /‰¥Ä"^¥∞!!
±¥@"’¥¿"#Y¥#ØATT000000003ATTO FastFrame N351N351∞ATT000000004ATTO FastFrame N352N352±ATT000000005ATTO FastFrame N311N311≤ATT000000006ATTO FastFrame N312N312¥ATT000000007ATTO FastFrame N322N322∏@ATT000000008ATTO ThunderLink N3 3102N3 3102∑@ATT000000009ATTO ThunderLink NS 3252NS 3252Mellanox Ethernet driver (3.1.0-dev) PËvHÄÓ∆ ËvH«2P‰T'Ä «2‰T'@«+P ö;ËÄR«+ ö;Ëq«"ˇˇˇˇ¿ã é‡éè∑¡n;Ç Ÿ&C
‹vkk≈≤MÜPG∏Ì&…"÷÷ä/aÀK+dõ5”ÜÕ1
L187321:
!pbase/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_fs_tree.c!devmlx5_core: %s:%s:%dpid %d): failed to allocate inbox
L187528:
mlx5_get_rsccreate_qprqsq_commonpxd_ready_bppci_write_bppci_read_bppci_read_stuck_no_completion_buffermax_pci_bwused_pci_bwrx_pci_errorstx_pci_errorstx_pci_correctable_errorstx_pci_non_fatal_errorstx_pci_fatal_errorsl0_mtt_missl0_mtt_hitl1_mtt_missl1_mtt_hitl0_mpt_missl0_mpt_hitl1_mpt_missl1_mpt_hitrxb_no_slow_path_creditsrxb_no_fast_path_creditsrxb_rxt_no_slow_path_cred_perf_countrxb_rxt_no_fast_path_cred_perf_countrxt_ctrl_perf_slice_load_slowrxt_ctrl_perf_slice_load_fastrxt_steering_perf_count_steering0_rse_work_raterxt_steering_perf_count_steering1_rse_work_rateperf_count_tpt_creditperf_wb_missperf_wb_hitrxw_perf_rx_l1_slow_miss_ldbrxw_perf_rx_l1_slow_hit_ldbrxw_perf_rx_l1_fast_miss_ldbrxw_perf_rx_l1_fast_hit_ldbrxw_perf_l2_cache_read_miss_ldbrxw_perf_l2_cache_read_hit_ldbrxw_perf_rx_l1_slow_miss_reqslrxw_perf_rx_l1_slow_hit_reqslrxw_perf_rx_l1_fast_miss_reqslrxw_perf_rx_l1_fast_hit_reqslrxw_perf_l2_cache_read_miss_reqslrxw_perf_l2_cache_read_hit_reqslrxs_no_pxt_creditsrxc_eq_all_slices_busyrxc_cq_all_slices_busyrxc_msix_all_slices_busysxw_qp_done_due_to_vl_limitedsxw_qp_done_due_to_deschedsxw_qp_done_due_to_work_donesxw_qp_done_due_to_limitedsxw_qp_done_due_to_e2e_creditssxw_packet_send_sxw2sxp_go_vldsxw_perf_count_steering_hitsxw_perf_count_steering_misssxw_perf_count_steering_rse_0sxd_no_sched_creditssxd_no_slow_path_sched_creditstpt_indirect_mem_keymlx5_alloc_db_from_pgdir(pgdir, db)/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_alloc.cHardwareLroHardware LROEnable hardware LRO.
L189117:
A˘÷PêA˘÷PêA˘÷PêA˘÷PêA˘÷PêA˘÷Pê"A˘÷Pê&A˘÷Pê*A˘÷Pê2A˘÷Pê6A˘÷Pê:A˘÷Pê>A˘÷PêFA˘÷PêJA˘÷PêNA˘÷PêRA˘÷PêVA˘÷PêZA˘÷Pê^A˘÷PêbA˘÷PêfA˘÷PêjA˘÷PênA˘÷PêrA˘÷PêvA˘÷PêzA˘÷Pê~A˘÷PêÇA˘÷PêÜA˘÷PêäA˘÷PêéA˘÷PêíA˘÷PêñA˘÷PêöA˘÷PêûA˘÷Pê¢A˘÷Pê¶A˘÷Pê™A˘÷PêÆA˘÷Pê≤A˘÷Pê∂A˘÷Pê∫A˘÷PêæA˘÷Pê¬A˘÷Pê∆A˘÷Pê A˘÷PêŒA˘÷Pê“A˘÷Pê÷A˘÷PêfiA˘÷Pê‚A˘÷PêÊA˘÷PêÍA˘÷PêÓA˘÷PêÚA˘÷PêˆA˘÷Pê˙A˘÷Pê˛A˘÷PêB˘÷ÓHOpÙPOp˙XOp `Op hOp pOp xOp ÄOp àOp„@ê ? êOp5 êBph òOp\ Cê÷@B0ÄB@
) –BpM PC0êC@—†C`ØATT000000003ATTO FastFrame N351N351∞ATT000000004ATTO FastFrame N352N352±ATT000000005ATTO FastFrame N311N311≤ATT000000006ATTO FastFrame N312N312¥ATT000000007ATTO FastFrame N322N322∏@ATT000000008ATTO ThunderLink N3 3102N3 3102∑@ATT000000009ATTO ThunderLink NS 3252NS 3252Mellanox Ethernet driver (3.1.0-dev) PËvHÄ@ ËvH;@2P‰T'ÄM@2‰T'm@+P ö;ËÄ@+ ö;ËûP"ˇˇˇˇ·#Äê„ÿsÄ<‰ÈÄl‰#@à∑¡n;Ç Ÿ&C
‹vkk≈≤MÜPG∏Ì&…"÷÷ä/aÀK+dõ5”ÜÕ1

I also found the ATTO FastFrame N322, which is a 2x25 GbE PCIe NIC. This may be the card that is used inside the ThunderLink product above. A better picture is available here.
If so, it does look very much like the Mellanox MCX4121A-ACAT: https://store.mellanox.com/products...25gbe-dual-port-sfp28-pcie3-0-x8-rohs-r6.html
I mean, all the components on the PCB have identical shape and placement (though I cannot read the exact writing on the small ICs).

Therefore I am almost certain that these 25+ GbE ATTO products are rebranded Mellanox (now Nvidia) products.
However, that does NOT mean that a Mellanox card will work directly with the ATTO driver. Mellanox/Nvidia does not officially supply a Mac driver (at least not from their downloads page), and it may well be that ATTO has made modifications to the firmware, just like Small Tree did with their Intel-based X520 10GbE NICs. It later did turn out that it was possible to modify standard Intel cards to be compatible with the Small Tree drivers, see this link.

Finally, if anyone has a Mellanox MCX4xxx card and could try it with the ATTO driver and give some feedback here that would be great!
Otherwise I think if you want 25/50/100 GbE on a Mac today I think the best option price-wise is to look for a used Chelsio T6225 / T6250 / T62100 card on eBay and put in a Thunderbolt enclosure. Chelsio provides a Mac driver that is supposed to be working very well.
If you need faster than 10G but can't afford 25/50/100G switches, then it may also be worth looking into the a bit older Chelsio T580 card. This is 2x40 GbE, so using QSFP+ instead of QSFP28. That could be a great fit together with one of the relatively cheap Mikrotik switches with 40G.
(Side note: Older Chelsio S3xx/T3xx/T4xx cards should still work well in a Mac Pro with PCIe slots, but supposedly do not support Thunderbolt tunneling. I've got an old Chelsio S310E-CR that I tried putting in a Akitio Node TB2 chassis. It is detected by the driver even in Big Sur, shows up in system settings, says a cable is connected, but cannot get a link and only gets a self-assigned IP address. Setting an IP manually did not help.)

EDIT: It seems MacRumors forum member nbritton has already done some work on modifying other Mellanox cards to work on the ATTO driver here. Nothing specifically mentioned about these 25G cards though.

 

[Kris Kelvin]

Otherwise I think if you want 25/50/100 GbE on a Mac today I think the best option price-wise is to look for a used Chelsio T6225 / T6250 / T62100 card on eBay and put in a Thunderbolt enclosure.

My experience with Chelsio cards have been mixed. Chelsio's drivers for macOS are usually out of date and Thunderbolt compatibility is also a potential issue – not only with the series you mentioned.

As for ATTO using Mellanox hardware, yes, that is indeed the case. However, modifying drivers is not something I'd recommend for professional use, where insufficient reliability can become expensive quite quickly.

I've tinkered enough in the past 20 years, so I'm more interested in a solution that works well out of the box. If only there was someone that could confirm that the ATTO adapter does that.

An alternative would be trunking 2 x 10Gbit, which probably wouldn't be slower – but definitely cheaper.

 

[TrumanLA]

My experience with Chelsio cards have been mixed. Chelsio's drivers for macOS are usually out of date and Thunderbolt compatibility is also a potential issue – not only with the series you mentioned.

As for ATTO using Mellanox hardware, yes, that is indeed the case. However, modifying drivers is not something I'd recommend for professional use, where insufficient reliability can become expensive quite quickly.

I've tinkered enough in the past 20 years, so I'm more interested in a solution that works well out of the box. If only there was someone that could confirm that the ATTO adapter does that.

An alternative would be trunking 2 x 10Gbit, which probably wouldn't be slower – but definitely cheaper.

Awesome! Mellanox is EXACTLY who I assumed was the OEM for ATTO NICs.
(That checkered aluminum heat dissipation design is fairly recognizable).

I realize this isn't exactly a new thread, but at the same time (unless you've posted on servethehome or something)
There just are NOT a lot of places where you're going to find a community with this kind of info (IMO).


FWIW:
I believe I've found a TOTALLY GHETTO FAB (& EXCEPTIONALLY inexpensive) method to get a TB3 to SFP28 device:

- Mellanox ConnectX-4 SFP28 (perhaps via the driver you reference at the end of that (absolute WALL of) text).
- Chelsio T520-SO-CR SFP28 (I have a couple of these).

via Thunderbolt 3

Instead of spending a freaking FORTUNE on that stupid ATTO 3252, my "idea" requires already having:
- Thunderbolt 3 to M.2 interface (I have the OWC EnvoyExpress // Sonnet Fusion // Samsung X5 interfaces to test w)
- SFP28 network adapter (I already have a couple of the above-mentioned Chelsio adapters)


I have a few interfaces I can test with in case the TB3 controller matters

Thunderbolt 3 to M.2 (NVMe):

• OWC EnvoyExpress
• Sonnet Fusion
• Samsung X5

And, the following NICs I can test:

• Mellanox ConnectX-3 QSFP+
• Chelsio T520-SO-CR SFP28 network adapters
• Chelsio SFP+ (older model-network adapters)
• also, considering purchasing a Mellanox SFP28

I'd only be able to test 'Peer to Peer' as I have only an SFP+ switch working atm (my SFP28 switch needs parts).

It's a "project" I bought off eBay for which I need some parts (fans and to install the OS) before I can use it.


Dell S5148F-ON:

48x SFP28 ( 25GbE)
6x QSFP28 (100GbE)

I think it needs 2-3 fans (over $125ea, IF I've even identified the right fans).

• Dell P/N: 3CH15 / 03CH15 (apparently the first '0' in the name can be ignored)
• Z9100-ON (which is the same as the above part)

But those PNs are assumptions as virtually no info exists as to which fans the S5148F-ON uses.

I only have the possible DPNs by watching a Serve The Home vid on YT from a few FRAMES showing a PN.

I had already tried:
Express Tag: Useless
Google-image search
Called Dell Ent. Support
Viewed eBay listings for S5148F component-pictures.
EVEN WITH the PN I couldn't find "supported devices" to ratify my assumption(s).

Anyway ... in the interim, I have some
- ATTO 10G-baseT to Thunderbolt
- ATTO SFP+ to Thunderbolt ...

I'm going to see if they use a PCIe interface and if I can just swap any of the:
- Mellanox QSFP+ (an old one)
- Chelsio SFP28 (I have two so I can at least test them).
- Chelsio SFP+
- Miricom SFP+ card

Of course -- I SUCK at networking ... so if anything gets complicated it'll be slow going, but I'll report back anything useful I discover, and any questions you have, just ask and I'll do my best.

My goal / plan is to use these to speed up access to some TrueNAS server performance.
While it'll do little if anything for the 8-bay T320 ... The:

Dell PowerEdge R730xd
- (4) PM983 3.84TB NVMe SSD
- (1) SLOG - 8GB RMS-200 Radian
- (1) L2Arc Optane P5800X 400GB
- (4) L1Arc 16GB PC4-2100 ECC (64GB)
- (1) SFP28 Dual-Port NDC (ConnectX-4)

After some testing I'll probably add 4-8 NVMe drives pending on controller options.

 

[TrumanLA]

My experience with Chelsio cards have been mixed. Chelsio's drivers for macOS are usually out of date and Thunderbolt compatibility is also a potential issue – not only with the series you mentioned.

As for ATTO using Mellanox hardware, yes, that is indeed the case. However, modifying drivers is not something I'd recommend for professional use, where insufficient reliability can become expensive quite quickly.

I've tinkered enough in the past 20 years, so I'm more interested in a solution that works well out of the box. If only there was someone that could confirm that the ATTO adapter does that.

An alternative would be trunking 2 x 10Gbit, which probably wouldn't be slower – but definitely cheaper.

Obviously there are things which can aggregate several 1GbE to increase the speed of a single file, but it's not what LAG does ... just your aggregate max speed ... when multiple things are being transferred.

How does it work with SFP+ ...?
If I'm not mistaken, SMB 3 or 4 or 5 or whatever ... offers a LAG which works on a single file..?

Had to find it bc I know it's the noob assumption ... and figured I'd head off those charges:

https://www.reddit.com/r/synology/comments/i9s83e
Still, is that more 'native' with SFP protocols ..?

 

[Kgeee]

Obviously there are things which can aggregate several 1GbE to increase the speed of a single file, but it's not what LAG does ... just your aggregate max speed ... when multiple things are being transferred.

How does it work with SFP+ ...?
If I'm not mistaken, SMB 3 or 4 or 5 or whatever ... offers a LAG which works on a single file..?

Had to find it bc I know it's the noob assumption ... and figured I'd head off those charges:

https://www.reddit.com/r/synology/comments/i9s83e
Still, is that more 'native' with SFP protocols ..?

"SMB Multichannel allows macOS to establish more than one connection to an SMB server, increase transfer speeds, and provide redundancy."

Configure SMB Multichannel behavior - Apple Support

In macOS Big Sur 11.3 and later, SMB Multichannel is enabled with default settings that are ideal for most organizations and users, but users can make adjustments as needed.

support.apple.com

I have absolutely no practical experience. I'm not even sure if I'm reading Apple's article right.

Sometimes I'd wish I was a Windows Dekstop user. Fairly cheap 100 GbE cars all day long on eBay.

 

[TrumanLA]

SMB Multichannel allows macOS to est. >1 SMB connection to a server, increasing xfer speeds & redundancy.

Configure SMB Multichannel behavior - Apple Support

In macOS Big Sur 11.3 and later, SMB Multichannel is enabled with default settings that are ideal for most organizations and users, but users can make adjustments as needed.

support.apple.com

Without practical experience who knows how legit it's dev stage is at ...ALPHA?

Sometimes I'd wish I was a Windows Dekstop user.
Fairly cheap 100 GbE cars all day long on eBay.

NO WAY. Just USE a freaking PC for a DAY OR TWO. Sure, if you ... never exit a GAME or something.
But if you want your shopping notes immediately sync'd to notes !?
Take a QR code to USPS (in notes) to have them just scan it and print it.
To have NEVER LOST A CONTACT in 12 YEARS !!!
To have struggled rebooting my iPhone ONE TIME (bc it happened so little, I didn't know how to do it.)

I don't if I COULD install a windows Print Driver!!! it's been that long since I've had to!

And 11? Has just gotten WORSE since 10.

Had I not finished my RAID recovery on a 6 drive GOOFY ass array w offsets ?? the stupid machine REBOOTED!!

And hoo hoo. I bought a PCIe enclosure, and I'm just going to see if it reads Mellenox SFP28

When I'm not slammed with ****, I'll let you know if it works. But uh, 40Gb/x (it REALLY should [ x s ]
UBIQUITOUS !??? I mean sheesus kariiiist ... just konw !?

The maanufacturing of thier **** is getting stupider and stupider,
I GRANT YOU soldered
RAM
GPA
CPU
1,000 1mm x 0.2mm screws to do a display (if they could they make them out of 0.25 x 0.5 µm and PROPRIETIZE THEM, too! aassRATS.

I wish there were an afterlife, just so I could watch Cooke get the everliving **** beat out of him by Jobs.

Anyway, thanks for the reply ... good info, and I'll get back with you if I get any info.

 

[TrumanLA]

Obviously there are things which can aggregate several 1GbE to increase the speed of a single file, but it's not what LAG does ... just your aggregate max speed ... when multiple things are being transferred.

How does it work with SFP+ ...?
If I'm not mistaken, SMB 3 or 4 or 5 or whatever ... offers a LAG which works on a single file..?

Had to find it bc I know it's the noob assumption ... and figured I'd head off those charges:

https://www.reddit.com/r/synology/comments/i9s83e
Still, is that more 'native' with SFP protocols ..?

MUCH more. The minimum you can get on a is 20Gb/s ... if you get lucky with SMBa MultiChannel ...

If you get a 2-port card, you can 20, 4-port, foty.

But, that's gonna be one loud ass switch if that's just ONE switch.

Anyone craving 80Gb x sec Thunderbolt switches .... no need for lagging ... just plugging in TB SSDs ...

with a HUGE drop in active thunderbolt because those might cost more than a house.

 

[Kgeee]

NO WAY. Just USE a freaking PC for a DAY OR TWO. Sure, if you ... never exit a GAME or something.
But if you want your shopping notes immediately sync'd to notes !?
Take a QR code to USPS (in notes) to have them just scan it and print it.
To have NEVER LOST A CONTACT in 12 YEARS !!!
To have struggled rebooting my iPhone ONE TIME (bc it happened so little, I didn't know how to do it.)

I don't if I COULD install a windows Print Driver!!! it's been that long since I've had to!

And 11? Has just gotten WORSE since 10.

Had I not finished my RAID recovery on a 6 drive GOOFY ass array w offsets ?? the stupid machine REBOOTED!!

And hoo hoo. I bought a PCIe enclosure, and I'm just going to see if it reads Mellenox SFP28

When I'm not slammed with ****, I'll let you know if it works. But uh, 40Gb/x (it REALLY should [ x s ]
UBIQUITOUS !??? I mean sheesus kariiiist ... just konw !?

The maanufacturing of thier **** is getting stupider and stupider,
I GRANT YOU soldered
RAM
GPA
CPU
1,000 1mm x 0.2mm screws to do a display (if they could they make them out of 0.25 x 0.5 µm and PROPRIETIZE THEM, too! aassRATS.

I wish there were an afterlife, just so I could watch Cooke get the everliving **** beat out of him by Jobs.

Anyway, thanks for the reply ... good info, and I'll get back with you if I get any info.

True. That's why I'm not switching even though network gear is so much less expensive.

Let me know if you ever get the 25G working on the PCIe enclosure. As you wrote earlier, there really isn't too much information about this topic out there. I guess the easiest method is just purchasing those ATTO Thunderbolt devices, but the price is just a bit too spicy.

 

[TrumanLA]

True. That's why I'm not switching even though network gear is so much less expensive.

Let me know if you ever get the 25G working on the PCIe enclosure. As you wrote earlier, there really isn't too much information about this topic out there. I guess the easiest method is just purchasing those ATTO Thunderbolt devices, but the price is just a bit too spicy.

Will do; here's my reply on another thread:
My ATTO 10GbE to TB not only works fine with 10.10 - 10.13 ... but also 10.14,
AND ... my 2019 MBP. Didn't even bother trying to install drivers. It's straight PnP

- 10.10 - 10.13
- 10.14 (tested after Mojave was just a "fuggit, why not check" w/o expectation.

In fact.., by then? I was using a TB3 to TB2 adapter to do so as I was on a 2019.

- Mojave .....then
- Catalina ...then
- Big Sur ....kept working.
- Monterey ..and yeah, even Monterey.

I have a sonnet now too, but mee so hooney for SFP28.
Bought a PCIe Riser to M2 Riser to connect to a portable M.2 interface as a PCIe proxy.

I don't know if others do, but I just think of TB as a compact PCIe 3.0 x4 connector.

PCIe (Melanox SFP28) --> to x4 Riser
Riser uses an M2 interface to transfer data
(which I have 3-4 different TB3 and 2-3 TB2 to try mixing and matching with)
(If +, I'll test via USB-C (10Gb) for budget solution for those younger readers).

From there, it's pseudo M.2 --> via M.2 to TB3.
I'll test as many as I can to report back which work and what doesn't.

I have:
- ~5 SFP+ cards
- ~3 SFP28 devs
- Sonnet Fusion (TB2)
- Sonnet Fusion (TB3)
- Sam X5 (TB3)
- OWC x2 (TB3)

I have duplicates of
- 5x - Sam X5 M2 --> TBx
- 5x - 10Gb SFP+ --> PCIe
- 3x - x4 PCI slot --> M2

So if it doesn't work, hopefully I'm able to rule out a defective cable (I have 3 of them) ... or not.

I'd've been done Tuesday, but they "generously" sent the "eGPU" slot (x16) which ≠ x4


Ordered 3x of the PCIe x4 + x1 (x1 works with x4) which'll all be here in a couple days.

I bet someone already knows why my boneheaded idea will work, but I don't; so experiments all I got.

eGPU boxes have gone UP !! I wish more companies made eSFP28+ or SFP100 to TB3.
ATTO's monopoly gifts then a 300% markup, anyway ...

All be back

 

[Kgeee]

Will do; here's my reply on another thread:
My ATTO 10GbE to TB not only works fine with 10.10 - 10.13 ... but also 10.14,
AND ... my 2019 MBP. Didn't even bother trying to install drivers. It's straight PnP

- 10.10 - 10.13
- 10.14 (tested after Mojave was just a "fuggit, why not check" w/o expectation.

In fact.., by then? I was using a TB3 to TB2 adapter to do so as I was on a 2019.

- Mojave .....then
- Catalina ...then
- Big Sur ....kept working.
- Monterey ..and yeah, even Monterey.

I have a sonnet now too, but mee so hooney for SFP28.
Bought a PCIe Riser to M2 Riser to connect to a portable M.2 interface as a PCIe proxy.

I don't know if others do, but I just think of TB as a compact PCIe 3.0 x4 connector.

PCIe (Melanox SFP28) --> to x4 Riser
Riser uses an M2 interface to transfer data
(which I have 3-4 different TB3 and 2-3 TB2 to try mixing and matching with)
(If +, I'll test via USB-C (10Gb) for budget solution for those younger readers).

From there, it's pseudo M.2 --> via M.2 to TB3.
I'll test as many as I can to report back which work and what doesn't.

I have:
- ~5 SFP+ cards
- ~3 SFP28 devs
- Sonnet Fusion (TB2)
- Sonnet Fusion (TB3)
- Sam X5 (TB3)
- OWC x2 (TB3)

I have duplicates of
- 5x - Sam X5 M2 --> TBx
- 5x - 10Gb SFP+ --> PCIe
- 3x - x4 PCI slot --> M2

So if it doesn't work, hopefully I'm able to rule out a defective cable (I have 3 of them) ... or not.

I'd've been done Tuesday, but they "generously" sent the "eGPU" slot (x16) which ≠ x4


Ordered 3x of the PCIe x4 + x1 (x1 works with x4) which'll all be here in a couple days.

I bet someone already knows why my boneheaded idea will work, but I don't; so experiments all I got.

eGPU boxes have gone UP !! I wish more companies made eSFP28+ or SFP100 to TB3.
ATTO's monopoly gifts then a 300% markup, anyway ...

All be back

Doing gods work. Really interested to see if it works. Thanks man.

 

[TrumanLA]

I'm REALLY hoping that the BDS kexts for these companies were adequate to show us is SFP28 is just baked in.

 

[DCswitch]

Has anyone tried a Chelsio card 40G or faster with macOS Monterey or Ventura? I have it working great in Big Sur on a Mac Pro 2019, but I couldn't get it to work in Monterey with their old Catalina driver (I was able to finagle Big Sur with a little tweaking). If I have to send it to an M.2 or TB adapter, I will, but it's kind of annoying.

I have written to Chelsio so many times through the years and they keep saying they are working on it, year after year after year they are apparently working on it with no release in sight.

 

[metacollin]

Has anyone tried a Chelsio card 40G or faster with macOS Monterey or Ventura? I have it working great in Big Sur on a Mac Pro 2019, but I couldn't get it to work in Monterey with their old Catalina driver

I have a T580-LP-CR that I am attempting to get to work with Ventura. The good news is, as old as the driver may be, it is still compatible and loads/boots just fine in Ventura (though it will complain about it having an expired certificate... its that old lol). No clue if it will work with Apple Silicon, I'm still rolling with x86_64.

It detects the card just fine but fails to actually initialize it and bitches about not getting enough MSI interrupts (I know the 'i' is for interrupt.. I just don't care. Atm machine) allocated (it wants 8 and thinks it only got 1).

This is either neutral news or bad news. It is bad news if it actually can't allocate more than one MSI interrupt due to something that has changed in newer versions of macOS (which, honestly, I can't imagine what could possibly cause that, but here we are).

The neutral news, which IMO is far more likely, is that its MSI detection code is written in a dumb way and it no longer correctly counts the number of interrupts which are MSI ones (in the macOS kernel doesn't have a direct way to locate them, you have to start at an address and iterate over all the interrupts and identify if they are MSI ones ore not) so the poor thing incorrectly thinks there is only 1 MSI interrupt left. There are like 2047 of these mofos and even a big phat NVMe drive might use maybe 65 max. There is no way the system is actually out of them, and except for exactly one.

So why is it neutral news? Because there is a chance I can patch the driver to be less stupid if it is something as simple as bypassing a broken check. Or it might be a non-starter, we'll see. It kind of defeats the purpose of the card if it can't use MSI interrupts.

I'll keep you guys updated. And have some faith - I'm running macOS with a stock kernel and native power management on a Super Micro motherboard thanks to my own BIOS patch, so this isn't my first rodeo. Or maybe dog and pony show? *shrug*

 

[metacollin]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).

No word on speed yet, but it works fine with cxgbtool at least.

 

[Kris Kelvin]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Impressive! Thanks for sharing!

I believe the kext could be re-signed with

codesign -f -s <identity> <path>

(Apple Developer account required). I currently don't have a Chelsio card to try this though.

 

[m1carbon]

I've been wanting to find out more about 25GbE options for Mac as well. I downloaded the driver for ThunderLink NS3252 and used The Unarchiver to extract some of the files from the ATTOThunderLinkNC3_1081f1.pkg.
Inside the ATTOThunderLinkNC3_1081f1/ATTOThunderLinkNC3.pkg/Payload file there are multiple mentions of Mellanox:

L186879: !pbase/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_fs_tree.c!devmlx5_core: %s:%s:%dpid %d): failed to allocate inbox
L186890:
fdb_rootfdb_prioesw_egress_rootesw_egress_prioesw_ingress_rootesw_ingress_priosniffer_tx_rootsniffer_priosniffer_rx_rootmlx5_cmd_fs_set_fte"queue_work on inactive work queue\n"@/Volumes/Development/mellanox/source/cna3/osx/kext/compat/osdep.cpp:174mlx5_core: %s:%s:%dpid %d): MLX5_CMD_OP_MODIFY_VPORT_STATE failed
L187086:
mlx5_get_rsccreate_qprqsq_commonpxd_ready_bppci_write_bppci_read_bppci_read_stuck_no_completion_buffermax_pci_bwused_pci_bwrx_pci_errorstx_pci_errorstx_pci_correctable_errorstx_pci_non_fatal_errorstx_pci_fatal_errorsl0_mtt_missl0_mtt_hitl1_mtt_missl1_mtt_hitl0_mpt_missl0_mpt_hitl1_mpt_missl1_mpt_hitrxb_no_slow_path_creditsrxb_no_fast_path_creditsrxb_rxt_no_slow_path_cred_perf_countrxb_rxt_no_fast_path_cred_perf_countrxt_ctrl_perf_slice_load_slowrxt_ctrl_perf_slice_load_fastrxt_steering_perf_count_steering0_rse_work_raterxt_steering_perf_count_steering1_rse_work_rateperf_count_tpt_creditperf_wb_missperf_wb_hitrxw_perf_rx_l1_slow_miss_ldbrxw_perf_rx_l1_slow_hit_ldbrxw_perf_rx_l1_fast_miss_ldbrxw_perf_rx_l1_fast_hit_ldbrxw_perf_l2_cache_read_miss_ldbrxw_perf_l2_cache_read_hit_ldbrxw_perf_rx_l1_slow_miss_reqslrxw_perf_rx_l1_slow_hit_reqslrxw_perf_rx_l1_fast_miss_reqslrxw_perf_rx_l1_fast_hit_reqslrxw_perf_l2_cache_read_miss_reqslrxw_perf_l2_cache_read_hit_reqslrxs_no_pxt_creditsrxc_eq_all_slices_busyrxc_cq_all_slices_busyrxc_msix_all_slices_busysxw_qp_done_due_to_vl_limitedsxw_qp_done_due_to_deschedsxw_qp_done_due_to_work_donesxw_qp_done_due_to_limitedsxw_qp_done_due_to_e2e_creditssxw_packet_send_sxw2sxp_go_vldsxw_perf_count_steering_hitsxw_perf_count_steering_misssxw_perf_count_steering_rse_0sxd_no_sched_creditssxd_no_slow_path_sched_creditstpt_indirect_mem_keymlx5_alloc_db_from_pgdir(pgdir, db)/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_alloc.cHardwareLroHardware LROEnable hardware LRO.
L187137:
œÌÒ
*ÒLÒmÒÄÒóÒÆÒ«Ò ÂÒ Ú Ú 8Ú WÚ vÚ íÚ ØÚ ÕÚ$‚Ú$Û(v¥–.|¥ÿ.Ç¥‡.à¥Ë.é¥..ö¥/†¥/¶¥/k¥p «¥/Ω¥"¥ /‰¥Ä"^¥∞!!
±¥@"’¥¿"#Y¥#ØATT000000003ATTO FastFrame N351N351∞ATT000000004ATTO FastFrame N352N352±ATT000000005ATTO FastFrame N311N311≤ATT000000006ATTO FastFrame N312N312¥ATT000000007ATTO FastFrame N322N322∏@ATT000000008ATTO ThunderLink N3 3102N3 3102∑@ATT000000009ATTO ThunderLink NS 3252NS 3252Mellanox Ethernet driver (3.1.0-dev) PËvHÄÓ∆ ËvH«2P‰T'Ä «2‰T'@«+P ö;ËÄR«+ ö;Ëq«"ˇˇˇˇ¿ã é‡éè∑¡n;Ç Ÿ&C
‹vkk≈≤MÜPG∏Ì&…"÷÷ä/aÀK+dõ5”ÜÕ1
L187321:
!pbase/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_fs_tree.c!devmlx5_core: %s:%s:%dpid %d): failed to allocate inbox
L187528:
mlx5_get_rsccreate_qprqsq_commonpxd_ready_bppci_write_bppci_read_bppci_read_stuck_no_completion_buffermax_pci_bwused_pci_bwrx_pci_errorstx_pci_errorstx_pci_correctable_errorstx_pci_non_fatal_errorstx_pci_fatal_errorsl0_mtt_missl0_mtt_hitl1_mtt_missl1_mtt_hitl0_mpt_missl0_mpt_hitl1_mpt_missl1_mpt_hitrxb_no_slow_path_creditsrxb_no_fast_path_creditsrxb_rxt_no_slow_path_cred_perf_countrxb_rxt_no_fast_path_cred_perf_countrxt_ctrl_perf_slice_load_slowrxt_ctrl_perf_slice_load_fastrxt_steering_perf_count_steering0_rse_work_raterxt_steering_perf_count_steering1_rse_work_rateperf_count_tpt_creditperf_wb_missperf_wb_hitrxw_perf_rx_l1_slow_miss_ldbrxw_perf_rx_l1_slow_hit_ldbrxw_perf_rx_l1_fast_miss_ldbrxw_perf_rx_l1_fast_hit_ldbrxw_perf_l2_cache_read_miss_ldbrxw_perf_l2_cache_read_hit_ldbrxw_perf_rx_l1_slow_miss_reqslrxw_perf_rx_l1_slow_hit_reqslrxw_perf_rx_l1_fast_miss_reqslrxw_perf_rx_l1_fast_hit_reqslrxw_perf_l2_cache_read_miss_reqslrxw_perf_l2_cache_read_hit_reqslrxs_no_pxt_creditsrxc_eq_all_slices_busyrxc_cq_all_slices_busyrxc_msix_all_slices_busysxw_qp_done_due_to_vl_limitedsxw_qp_done_due_to_deschedsxw_qp_done_due_to_work_donesxw_qp_done_due_to_limitedsxw_qp_done_due_to_e2e_creditssxw_packet_send_sxw2sxp_go_vldsxw_perf_count_steering_hitsxw_perf_count_steering_misssxw_perf_count_steering_rse_0sxd_no_sched_creditssxd_no_slow_path_sched_creditstpt_indirect_mem_keymlx5_alloc_db_from_pgdir(pgdir, db)/Volumes/Development/mellanox/source/cna3/osx/driver/mlx5/mlx5_core/mlx5_alloc.cHardwareLroHardware LROEnable hardware LRO.
L189117:
A˘÷PêA˘÷PêA˘÷PêA˘÷PêA˘÷PêA˘÷Pê"A˘÷Pê&A˘÷Pê*A˘÷Pê2A˘÷Pê6A˘÷Pê:A˘÷Pê>A˘÷PêFA˘÷PêJA˘÷PêNA˘÷PêRA˘÷PêVA˘÷PêZA˘÷Pê^A˘÷PêbA˘÷PêfA˘÷PêjA˘÷PênA˘÷PêrA˘÷PêvA˘÷PêzA˘÷Pê~A˘÷PêÇA˘÷PêÜA˘÷PêäA˘÷PêéA˘÷PêíA˘÷PêñA˘÷PêöA˘÷PêûA˘÷Pê¢A˘÷Pê¶A˘÷Pê™A˘÷PêÆA˘÷Pê≤A˘÷Pê∂A˘÷Pê∫A˘÷PêæA˘÷Pê¬A˘÷Pê∆A˘÷Pê A˘÷PêŒA˘÷Pê“A˘÷Pê÷A˘÷PêfiA˘÷Pê‚A˘÷PêÊA˘÷PêÍA˘÷PêÓA˘÷PêÚA˘÷PêˆA˘÷Pê˙A˘÷Pê˛A˘÷PêB˘÷ÓHOpÙPOp˙XOp `Op hOp pOp xOp ÄOp àOp„@ê ? êOp5 êBph òOp\ Cê÷@B0ÄB@
) –BpM PC0êC@—†C`ØATT000000003ATTO FastFrame N351N351∞ATT000000004ATTO FastFrame N352N352±ATT000000005ATTO FastFrame N311N311≤ATT000000006ATTO FastFrame N312N312¥ATT000000007ATTO FastFrame N322N322∏@ATT000000008ATTO ThunderLink N3 3102N3 3102∑@ATT000000009ATTO ThunderLink NS 3252NS 3252Mellanox Ethernet driver (3.1.0-dev) PËvHÄ@ ËvH;@2P‰T'ÄM@2‰T'm@+P ö;ËÄ@+ ö;ËûP"ˇˇˇˇ·#Äê„ÿsÄ<‰ÈÄl‰#@à∑¡n;Ç Ÿ&C
‹vkk≈≤MÜPG∏Ì&…"÷÷ä/aÀK+dõ5”ÜÕ1

I also found the ATTO FastFrame N322, which is a 2x25 GbE PCIe NIC. This may be the card that is used inside the ThunderLink product above. A better picture is available here.
If so, it does look very much like the Mellanox MCX4121A-ACAT: https://store.mellanox.com/products...25gbe-dual-port-sfp28-pcie3-0-x8-rohs-r6.html
I mean, all the components on the PCB have identical shape and placement (though I cannot read the exact writing on the small ICs).

Therefore I am almost certain that these 25+ GbE ATTO products are rebranded Mellanox (now Nvidia) products.
However, that does NOT mean that a Mellanox card will work directly with the ATTO driver. Mellanox/Nvidia does not officially supply a Mac driver (at least not from their downloads page), and it may well be that ATTO has made modifications to the firmware, just like Small Tree did with their Intel-based X520 10GbE NICs. It later did turn out that it was possible to modify standard Intel cards to be compatible with the Small Tree drivers, see this link.

Finally, if anyone has a Mellanox MCX4xxx card and could try it with the ATTO driver and give some feedback here that would be great!
Otherwise I think if you want 25/50/100 GbE on a Mac today I think the best option price-wise is to look for a used Chelsio T6225 / T6250 / T62100 card on eBay and put in a Thunderbolt enclosure. Chelsio provides a Mac driver that is supposed to be working very well.
If you need faster than 10G but can't afford 25/50/100G switches, then it may also be worth looking into the a bit older Chelsio T580 card. This is 2x40 GbE, so using QSFP+ instead of QSFP28. That could be a great fit together with one of the relatively cheap Mikrotik switches with 40G.
(Side note: Older Chelsio S3xx/T3xx/T4xx cards should still work well in a Mac Pro with PCIe slots, but supposedly do not support Thunderbolt tunneling. I've got an old Chelsio S310E-CR that I tried putting in a Akitio Node TB2 chassis. It is detected by the driver even in Big Sur, shows up in system settings, says a cable is connected, but cannot get a link and only gets a self-assigned IP address. Setting an IP manually did not help.)

EDIT: It seems MacRumors forum member nbritton has already done some work on modifying other Mellanox cards to work on the ATTO driver here. Nothing specifically mentioned about these 25G cards though.

So I tried my to mess with my server's Mellanox MCX4121A-ACAT, and it works on my M2 MBP 16" with the unmodified ATTO N322 drivers, but with some big drawbacks. I cannot change the MTU to 9000 and I think because of that, my speedtest speeds are 50m down, 1g up on my ATT 5g fiber service.. Installation of drivers was a breeze, but you do have to lower the security settings of your Mac.

I think maybe with further modification with the kext/fw/etc, we may get this fully functional, but I don't want to mess with this NIC. My Intel x520-DA2 is on its way, and that one I will mess around for sure.

EDIT: Forgot to mention that the Mellanox MCX412A-ACAT is connected using an OWC Helios 3S.

 

[DCswitch]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).

No word on speed yet, but it works fine with cxgbtool at least.

I'm having a bit of trouble getting the Chelsio working. I have it in a Mac Pro 7,1 (2019) running Ventura. I disabled SIP. I install the kext (I installed direct from Chelsio because I don't see a difference from the one you uploaded) and I tried to kextload, but it's just not working. Any thoughts?

 

[Kris Kelvin]

I installed direct from Chelsio because I don't see a difference from the one you uploaded

The uploaded kext IS different from the one provided by Chelsio; you can see that if you compare the cxgb binary in the kext package with a suitable tool (e.g. Hex Fiend).

If you'll try again with the uploaded kext, please report your findings.

 

[DCswitch]

The uploaded kext IS different from the one provided by Chelsio; you can see that if you compare the cxgb binary in the kext package with a suitable tool (e.g. Hex Fiend).

If you'll try again with the uploaded kext, please report your findings.

I originally did and I had no luck, so then I tried the original from Chelsio. It may have been an ownership issue, though.

1) copy the kext over
2) make sure ownership is correct
3) kextload it
4) any other instruction before I try again?

 

[DCswitch]

The uploaded kext IS different from the one provided by Chelsio; you can see that if you compare the cxgb binary in the kext package with a suitable tool (e.g. Hex Fiend).

If you'll try again with the uploaded kext, please report your findings.

Still no luck. I set permissions properly and nothing and SIP is disabled. The card is seen in System Report, but that's it. The driver does not load. I did not try this,
codesign -f -s <identity> <path>
So, not sure if that's the issue.

Any thoughts?

 

[TrumanLA]

True. That's why I'm not switching even though network gear is so much less expensive.

Let me know if you ever get the 25G working on the PCIe enclosure. As you wrote earlier, there really isn't too much information about this topic out there. I guess the easiest method is just purchasing those ATTO Thunderbolt devices, but the price is just a bit too spicy.

I got mac to recognize a Myllenox ConnectX-4 using an ATTO SFP28 cards ROM.

I think that's probably the least expensive / easiest solution (re-writing drivers sounds like a headache).

Unfortunately, Mojave just does not work with anything really ... I've tried:

- ATTO SFP28 card - Limited to 1Gb performance.
- Myllenox SFP28 - Same thing
- Chelsio T320 - Gets 10Gb but lost connection after 10GB.

Spoke to ATTO tech for an eternity and he basically said that Mojave's SMB stack is dog crap.
This pretty much means that unless you use OpenCORE with Catalina you're getting terrible performance.

I'm pretty confident that the ConnectX-4 via TB3 will be an excellent low-cost solution for SFP28 access.
But ... this is yet another nail in the cMP's coffin unfortunately.

Still, that the ConnectX-4 is so easily modified (to me) is outstanding.

 

[TrumanLA]

So I tried my to mess with my server's Mellanox MCX4121A-ACAT, and it works on my M2 MBP 16" with the unmodified ATTO N322 drivers, but with some big drawbacks. I cannot change the MTU to 9000 and I think because of that, my speedtest speeds are 50m down, 1g up on my ATT 5g fiber service.. Installation of drivers was a breeze, but you do have to lower the security settings of your Mac.

I think maybe with further modification with the kext/fw/etc, we may get this fully functional, but I don't want to mess with this NIC. My Intel x520-DA2 is on its way, and that one I will mess around for sure.

EDIT: Forgot to mention that the Mellanox MCX412A-ACAT is connected using an OWC Helios 3S.

The Mellanox MCX4121A-ACAT that you got working ... is that with the OEM Myllenox ROM or did you swap the ROM out for an ATTO ROM..? Also, I tried using both the ATTO and ConnectX-4 on a MP5,1 ... as well as a Chelsio S320E-SR and while the ATTO and ConnectX-4 were both limited to about 1Gb for me also (in the MP5,1) ... ATTO said it's bc Mojave's SMB stack is crap. But ... the S320E actually did give me 10GbE ... for 10Gb and then it **** the bed and lost connectivity. I tried installing my T520-SO ... but the cMP wouldn't even chime (post) with the card installed (in any PCIe slot). Very goofy considering Chelsio makes a driver that supposedly works with everything from El Capitan thru Mojave and a 2nd driver for Catalina (and presumably newer OS also). I'm waiting for a power cable to test these via TB3 with an OEM Myllenox, ATTO-ROM Myllenox, ATTO SFP28 card ... to see what really works etc. I don't think I'll bother checking the diff between 10Gb and SFP28 for now (I hate turning on my SFP28 switch bc it's much louder than the Mikrotik).

I don't imagine [how] the MCX412A is connected (PCIe vs TB3) really matters ... right..?

Also, re MTU 9000 ... I went through a thread on STH a couple of years ago by a guy who documented a crusade he'd gone on trying to get all of his gear working with MTU 9000 settings and ultimately found it to be more effort than it was worth. Did you come to a different conclusion..?

Thanks for the info you posted.

 

[DCswitch]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).

No word on speed yet, but it works fine with cxgbtool at least.

I'm wondering if you can help me out. I also made a Hackintosh using Supermicro with the X11SPA-T (customized firmware I had to get authenticated with Supermicro) and I did this prior to prices dropping on used Mac Pro 7,1's (2019). So, I ended up turning that X11 into my new server board on my 24 bay. I also have other Hackintoshes, but I would ideally like to get this going in the 7,1 and with an enclosure with my Mac Studio.

Is there anything more you can share? I was able to get Chelsio's driver to work in Big Sur, but no further than that. I am now running Ventura on most of my systems, so I would really like to get it working in Ventura. I have messaged Chelsio quite a bit for years and they keep saying they will release a new driver for macOS, but clearly those are just words as years have gone by and their last official driver is for Catalina.

Any thoughts for me? I have SIP disabled, I set permissions properly, the card is seen in the PCI section of System Report, but I cannot kextload the driver. I have no trouble with all the same hardware running Big Sur. Perhaps, I should try your kext in Big Sur and then do a migration into Ventura, but that seems unneccessary as I believe this can be solved if you've got it running in your Hack.

Is there anything else that OpenCore may be contributing or something else you might be overlooking as to why this wouldn't work in a genuine 7,1? I have all security turned off in Recovery besides SIP.

I don't like ATTO, OWC, or Sonnet for network cards and prefer Chelsio the most (I have my reasons I don't feel like getting into right now).

EDITed - I want to add that I also tried this in Recovery Terminal (see screenshot):
spctl kext-consent add 6DVCB76QJW

(Chelsio Developer ID 6DVCB76QJW)