MP All Models Mellanox ConnectX-3 40 GbE using ATTO FastFrame macOS driver

[nbritton]

Hi,

Did anybody else notice that the ATTO FastFrame NQ41 and NQ42 are rebranded Mellanox ConnectX-3 CX353A and CX354A cards? ATTO provides the bin firmware files on their website, all you likely need to do to get a generic Mellanox CX3 card to work is cross flash its firmware with ATTO's bin file. This is easy with mst / flint using Mellanox's Firmware Tools package.

1. Download Mellanox Firmware Tools: https://www.mellanox.com/products/adapter-software/firmware-tools
2. Download ATTO NQ41/NQ42 Firmware: https://www.atto.com/downloads/174/driver/29300356
3. Run the following:

mst start;

# NQ41 - Single Port Card
flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i flsbundle_nq41_2017_09_05.bin burn;

# NQ42 - Dual Port Card
flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i flsbundle_nq42_2017_09_05.bin burn;

4. Put device in a Mac and install ATTO's FastFrame macOS drivers.

I haven't tested this yet but it should work just fine, I cross flash Dell branded Mellanox cards all the time back to stock Mellanox firmware. I don't have a QSFP cable that is long enough to reach my Mac, but as soon as I do I'll flash one of mine and report back. CX3 cards can be picked up on eBay for about $40, the part numbers you should be looking for are CX353A-FCBT or CX354A-FCBT, these support up to 56 GbE. The CX353A-QCBT or CX354A-QCBT may also work, these support up to 40 GbE. At this time, the ATTO driver appears to only work with ConnectX-3 Pro CX313A and CX314A cards; the supported device id type (i.e. chipset model) is 4103. MCX313A-BCBT and MCX314A-BCBT are presumed to not be compatible because their device id is reportedly 4099.

 

[nbritton]

It appears the ATTO cards are device id 4103, which are ConnectX-3 Pro devices.

root@r920-cmwhv52:~# flint -i flsbundle_nq41_2017_09_05.bin q
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050
MACs:                                       0002c9000001     0002c9000002
VSD:                   n/a
PSID:                  ATT1060111023

root@r920-cmwhv52:~# flint -d /dev/mst/mt4099_pci_cr0 q
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 506b4b03004cfac0 506b4b03004cfac1 506b4b03004cfac2 506b4b03004cfac3
MACs:                                       506b4b4cfac1     506b4b4cfac2
VSD:
PSID:                  DEL1100001019

root@r920-cmwhv52:~# flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i flsbundle_nq41_2017_09_05.bin burn;

    Current FW version on flash:  2.42.5000
    New FW version:               2.42.5000

    Note: The new FW version is the same as the current FW version on flash.

 Do you want to continue ? (y/n) [n] : y


    You are about to replace current PSID on flash - "DEL1100001019" with a different PSID - "ATT1060111023".
    Note: It is highly recommended not to change the PSID.

 Do you want to continue ? (y/n) [n] : y
-E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only

 

[nbritton]

Stumbleine:Downloads root# pkgutil --expand ATTOFastFrame2_1110f1.pkg ./ATTOFastFrame2

Stumbleine:Downloads root# cd ATTOFastFrame2

Stumbleine:ATTOFastFrame2.pkg root# ls -l
total 1488
-rw-r--r--  1 root  wheel   36340 Sep 12  2018 Bom
-rw-r--r--@ 1 root  wheel     652 Sep 12  2018 PackageInfo
-rw-r--r--  1 root  wheel  715930 Sep 12  2018 Payload
drwxr-xr-x  4 root  wheel     128 Mar  7 09:52 Scripts

Stumbleine:ATTOFastFrame2.pkg root# file *
Bom:         Mac OS X bill of materials (BOM) file
PackageInfo: ASCII text
Payload:     gzip compressed data, from Unix, original size 2054144
Scripts:     directory

Stumbleine:ATTOFastFrame2.pkg root# mv Payload Payload.gz

Stumbleine:ATTOFastFrame2.pkg root# gunzip Payload.gz

Stumbleine:ATTOFastFrame2.pkg root# ls -l
total 4096
-rw-r--r--  1 root  wheel    36340 Sep 12  2018 Bom
-rw-r--r--@ 1 root  wheel      652 Sep 12  2018 PackageInfo
-rw-r--r--  1 root  wheel  2054144 Sep 12  2018 Payload
drwxr-xr-x  4 root  wheel      128 Mar  7 09:52 Scripts

Stumbleine:ATTOFastFrame2.pkg root# file *
Bom:         Mac OS X bill of materials (BOM) file
PackageInfo: ASCII text
Payload:     ASCII cpio archive (pre-SVR4 or odc)
Scripts:     directory

Stumbleine:ATTOFastFrame2.pkg root# mv Payload Payload.cpio

Stumbleine:ATTOFastFrame2.pkg root# cpio -idv < Payload.cpio
.
./usr
./usr/local
./usr/local/libexec
./usr/local/libexec/ATTOFastFrame2_tmp
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/_CodeSignature
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/_CodeSignature/CodeResources
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/MacOS
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/MacOS/ATTOFastFrame2
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/Info.plist
./Applications
./Applications/ATTO
./Applications/ATTO/FastFrame2
./Applications/ATTO/FastFrame2/Uninstall.command
./Applications/ATTO/FastFrame2/atflash2
./Applications/ATTO/FastFrame2/ReadMe.htm
./Applications/ATTO/FastFrame2/atnicnvr
./Applications/ATTO/FastFrame2/atnetstat
./Applications/ATTO/FastFrame2/atinfo
4012 blocks

 

[nbritton]

root@r920-cmwhv52:~# flint -i flsbundle_nq41_2017_09_05.bin dc
;; Generated automatically by iniprep tool on Tue Sep 05 14:05:13 IDT 2017 from ./cx3pro_MCX313A_40g_2.prs
;;
;; PRS  FILE FOR KESTREL BENTAL
;; $Id$



[PS_INFO]
Name = 4779-313A-X00_Ax
Description = ConnectX-3 Pro EN network interface card; 40/56GbE; single-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6
PRS_name    = cx3pro_MCX313A_40g_2.prs

[ADAPTER]
PSID = ATT1060111023
pcie_gen2_speed_supported = true
pcie_gen3_speed_supported = true
adapter_dev_id = 0x1007
silicon_rev = 0x00

gpio_mode1 = 0x08000001
gpio_mode0 = 0x04c04032
gpio_default_val = 0x0f306023
gpio_pull_up = 0xff2baf2f
gpio_pull_enable = 0xfbbbbfff

receiver_detect_en = true
vdd_change_to_1_offset = 5

nv_cfg_en = true

nv_config_sectors = 2

[HCA]
hca_header_subsystem_vendor_id = 0x117c
hca_header_device_id = 0x1007
hca_header_subsystem_id = 0x0090
hca_header_class_code = 0x020000
eth_xfi_en = true
mdio_en_port1 = 0
pcie_tx_polarity = 0x0f
dpdp_en = false

[IB]
mlpn_en_port0 = true
phy_type_port1 = XFI
ext_phy_board_port1 = FALCON
gen_guids_from_mac = true
do_sense = false
ref_clk_to_use = 0
module_power_level_supported_port0 = 5

num_of_ports = One_Port
new_gpio_scheme_en = true
read_cable_params_port1_en = true

cx3_spec1_3_ib_support_port0 = true
cx3_spec1_2_ib_support_port0 = true
spec1_3_fdr14_ib_support_port0 = true
spec1_3_fdr10_ib_support_port0 = true
mellanox_ddr_ib_support        = true
mellanox_qdr_ib_support        = true

port1_802_3ap_cr4_enable = true
port1_802_3ap_cr4_ability = true
port1_802_3ap_56kr4_ability = true

center_mix90phase = true

;;Logic lane to Serdes mapping
tx_logic_0_serdes = 0
tx_logic_1_serdes = 1
tx_logic_2_serdes = 2
tx_logic_3_serdes = 3
rx_logic_0_serdes = 3
rx_logic_1_serdes = 2
rx_logic_2_serdes = 1
rx_logic_3_serdes = 0

eth_tx_lane_polarity_port1 = 0xf
eth_rx_lane_polarity_port1 = 0x0
tx_lane_polarity_port1 = 0xf

; start of '#include "include_QSFP_serdes_prams_bental.h"'

;;Serdes parameters
port0_nego_fdr_mask_en = 0xfffc
port1_nego_fdr_mask_en = 0xfffc
port0_nego_fdr10_mask_en = 0xfffc
port1_nego_fdr10_mask_en = 0xfffc

nego_rx4_slicer_ind_en = 255
nego_rx4_slicer1_enable = 8
nego_rx4_slicer2_enable = 8
nego_rx4_ffe_tap0 = 94
nego_rx4_ffe_tap1 = 134
nego_rx4_ffe_tap2 = 245
nego_rx4_ffe_tap3 = 135
nego_rx4_ffe_tap4 = 171

nego_rx9_ffe_tap0=84
nego_rx9_ffe_tap1=164
nego_rx9_ffe_tap2=251
nego_rx9_ffe_tap3=132
nego_rx9_ffe_tap4=140

nego_rx15_ffe_tap3 = 140
nego_rx15_ffe_tap1 = 140

nego_rx10_ffe_tap3 = 140
nego_rx10_ffe_tap1 = 140

nego_rx8_ffe_tap3 = 140
nego_rx8_ffe_tap1 = 140

force_rx0_slicer_ind_en = 0x0
force_rx0_slicer1_enable = 0x0
force_rx0_slicer2_enable = 0x0
force_rx0_ffe_tap0 = 0xff
force_rx0_ffe_tap1 = 0x80
force_rx0_ffe_tap2 = 0x80
force_rx0_ffe_tap3 = 0x80
force_rx0_ffe_tap4 = 0x80

force_tx0_ob_preemp_pre = 0x40
force_tx0_ob_preemp_post = 0x0
force_tx0_ob_preemp_main = 0x7f
force_tx0_preemp = 0x0
force_tx0_pre_polarity = 0x1
force_tx0_post_polarity = 0x1
force_tx0_main_polarity = 0x0

force_rx2_slicer_ind_en = 0xeb
force_rx2_slicer1_enable = 0x0
force_rx2_slicer2_enable = 0x0
force_rx2_ffe_tap0 = 0x64
force_rx2_ffe_tap1 = 0x80
force_rx2_ffe_tap2 = 0xde
force_rx2_ffe_tap3 = 0x80
force_rx2_ffe_tap4 = 0x46

force_tx2_ob_preemp_pre = 0x30
force_tx2_ob_preemp_post = 0x0
force_tx2_ob_preemp_main = 0x7f
force_tx2_preemp = 0x0
force_tx2_pre_polarity = 0x1
force_tx2_post_polarity = 0x1
force_tx2_main_polarity = 0x0

force_rx3_slicer_ind_en = 0xff
force_rx3_slicer1_enable = 0x8
force_rx3_slicer2_enable = 0x8
force_rx3_ffe_tap0 = 0x6c
force_rx3_ffe_tap1 = 0x80
force_rx3_ffe_tap2 = 0xff
force_rx3_ffe_tap3 = 0x80
force_rx3_ffe_tap4 = 0x80

force_tx3_ob_preemp_pre = 0xc
force_tx3_ob_preemp_post = 0x7f
force_tx3_ob_preemp_main = 0x45
force_tx3_preemp = 0x0
force_tx3_pre_polarity = 0x1
force_tx3_post_polarity = 0x0
force_tx3_main_polarity = 0x1
force_tx3_ob_bias = 0xa

auto_ddr_tx_options = 2
auto_ddr_rx_options = 1

auto_qdr_tx_options = 6
auto_qdr_rx_options = 7

preset_tx_fdr_set12_ob_preemp_pre = 17
preset_tx_fdr_set12_ob_preemp_post = 0
preset_tx_fdr_set12_ob_preemp_main=25
preset_tx_fdr_set12_preemp = 0
preset_tx_fdr_set12_pre_polarity = 1
preset_tx_fdr_set12_post_polarity = 1
preset_tx_fdr_set12_main_polarity = 0
preset_tx_fdr_set12_ob_bias = 5

preset_tx_fdr_set13_ob_preemp_main =40
preset_tx_fdr_set13_ob_preemp_pre = 28
preset_tx_fdr_set13_ob_preemp_post = 0
preset_tx_fdr_set13_preemp = 0
preset_tx_fdr_set13_pre_polarity = 1
preset_tx_fdr_set13_post_polarity = 1
preset_tx_fdr_set13_main_polarity = 0
preset_tx_fdr_set13_ob_bias = 5

preset_tx_fdr_set14_ob_preemp_main = 35
preset_tx_fdr_set14_ob_preemp_pre = 25
preset_tx_fdr_set14_ob_preemp_post = 0
preset_tx_fdr_set14_preemp = 0
preset_tx_fdr_set14_pre_polarity = 1
preset_tx_fdr_set14_post_polarity = 1
preset_tx_fdr_set14_main_polarity = 0
preset_tx_fdr_set14_ob_bias = 5

preset_tx_fdr_set15_ob_preemp_main = 30
preset_tx_fdr_set15_ob_preemp_pre = 20
preset_tx_fdr_set15_ob_preemp_post = 0
preset_tx_fdr_set15_preemp = 0
preset_tx_fdr_set15_pre_polarity = 1
preset_tx_fdr_set15_post_polarity = 1
preset_tx_fdr_set15_main_polarity = 0
preset_tx_fdr_set15_ob_bias = 5

preset_tx_mask = 0xfffe

aba_mask0_start = 0
aba_mask0_end   = 3
aba_mask0 = 0x1000
aba_mask1_start = 4
aba_mask1_end   = 5
aba_mask1 = 0x8000
aba_mask2_start = 6
aba_mask2_end   = 10
aba_mask2 = 0x4000
aba_mask3_start = 11
aba_mask3_end   = 16
aba_mask3 = 0x2000

; ABA 40GE
aba_tx2_ob_preemp_pre = 20
aba_tx2_ob_preemp_main = 42
aba_tx2_ob_preemp_post = 8
aba_tx2_ob_bias = 8
aba_tx2_pre_polarity = 1
aba_tx2_post_polarity = 1
aba_tx2_main_polarity = 0

;;3m
aba_tx3_ob_preemp_pre = 22
aba_tx3_ob_preemp_main = 42
aba_tx3_ob_preemp_post = 5
aba_tx3_ob_bias = 8
aba_tx3_pre_polarity = 1
aba_tx3_post_polarity = 1
aba_tx3_main_polarity = 0

aba_tx4_ob_preemp_pre = 26
aba_tx4_ob_preemp_main = 42
aba_tx4_ob_preemp_post = 3
aba_tx4_ob_bias = 8
aba_tx4_pre_polarity = 1
aba_tx4_post_polarity = 1
aba_tx4_main_polarity = 0

aba_tx5_ob_preemp_pre = 60
aba_tx5_ob_preemp_main = 90
aba_tx5_ob_preemp_post = 8
aba_tx5_ob_bias = 8
aba_tx5_pre_polarity = 1
aba_tx5_post_polarity = 1
aba_tx5_main_polarity = 0

aba_tx6_ob_preemp_pre = 80
aba_tx6_ob_preemp_main = 110
aba_tx6_ob_preemp_post = 10
aba_tx6_ob_bias = 8
aba_tx6_pre_polarity = 1
aba_tx6_post_polarity = 1
aba_tx6_main_polarity = 0

aba_tx7_ob_preemp_pre = 75
aba_tx7_ob_preemp_main = 110
aba_tx7_ob_preemp_post = 15
aba_tx7_ob_bias = 8
aba_tx7_pre_polarity = 1
aba_tx7_post_polarity = 1
aba_tx7_main_polarity = 0

aba_fdr_tx16_ob_preemp_pre = 17
aba_fdr_tx16_ob_preemp_post = 0
aba_fdr_tx16_ob_preemp_main=25
aba_fdr_tx16_preemp = 0
aba_fdr_tx16_pre_polarity = 1
aba_fdr_tx16_post_polarity = 1
aba_fdr_tx16_main_polarity = 0
aba_fdr_tx16_ob_bias = 5

aba_fdr_tx17_ob_preemp_main =46
aba_fdr_tx17_ob_preemp_pre = 32
aba_fdr_tx17_ob_preemp_post = 0
aba_fdr_tx17_preemp = 0
aba_fdr_tx17_pre_polarity = 1
aba_fdr_tx17_post_polarity = 1
aba_fdr_tx17_main_polarity = 0
aba_fdr_tx17_ob_bias = 3

aba_fdr_tx18_ob_preemp_main = 50
aba_fdr_tx18_ob_preemp_pre = 32
aba_fdr_tx18_ob_preemp_post = 0
aba_fdr_tx18_preemp = 0
aba_fdr_tx18_pre_polarity = 1
aba_fdr_tx18_post_polarity = 1
aba_fdr_tx18_main_polarity = 0
aba_fdr_tx18_ob_bias = 3

aba_fdr_tx19_ob_preemp_main = 60
aba_fdr_tx19_ob_preemp_pre = 30
aba_fdr_tx19_ob_preemp_post = 0
aba_fdr_tx19_preemp = 0
aba_fdr_tx19_pre_polarity = 1
aba_fdr_tx19_post_polarity = 1
aba_fdr_tx19_main_polarity = 0
aba_fdr_tx19_ob_bias = 3

aba_index0_start = 0
aba_index0_end   = 3
aba_index0 = 0
aba_index1_start = 4
aba_index1_end   = 5
aba_index1 = 3
aba_index2_start = 6
aba_index2_end   = 9
aba_index2 = 2
aba_index3_start = 10
aba_index3_end   = 16
aba_index3 = 1

aba_rx2_slicer_ind_en = 0xeb
aba_rx2_slicer1_enable = 0x0
aba_rx2_slicer2_enable = 0x0
aba_rx2_ffe_tap0 = 0x80
aba_rx2_ffe_tap1 = 0x68
aba_rx2_ffe_tap2 = 0xd7
aba_rx2_ffe_tap3 = 0x80
aba_rx2_ffe_tap4 = 0x5a

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;; SFP+ section. all QSFP can be converted to SFP+ using QSA adapter.;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; ETH connected to third party device
aba_non_mlpn_tx8_ob_preemp_pre = 5
aba_non_mlpn_tx8_ob_preemp_post = 0
aba_non_mlpn_tx8_ob_preemp_main = 65
aba_non_mlpn_tx8_ob_bias = 8
aba_non_mlpn_tx8_pre_polarity = 1
aba_non_mlpn_tx8_post_polarity = 1
aba_non_mlpn_tx8_main_polarity = 0
aba_non_mlpn_tx8_preemp = 0


nego_eth_rx12_slicer_ind_en = 0xff
nego_eth_rx12_slicer1_enable= 0x8
nego_eth_rx12_slicer2_enable= 0x8
nego_eth_rx12_ffe_tap0=241
nego_eth_rx12_ffe_tap1=128
nego_eth_rx12_ffe_tap2=61
nego_eth_rx12_ffe_tap3=99
nego_eth_rx12_ffe_tap4=128

; end of '#include "include_QSFP_serdes_prams_bental.h"'

[PLL]
lbist_en  = 0
lbist_shift_freq  = 3
flash_div = 0x3
lbist_array_bypass = 1
lbist_pat_cnt_lsb = 0x2
core_f = 60
core_r = 14
core_od = 2
en_427_mhz = true

[FW]
flash_has_suspend_resume = 0
log_flashdev_size = 21
log_flash_sector_size = 2

root@r920-cmwhv52:~# flint -i flsbundle_nq42_2017_09_05.bin dc
;; Generated automatically by iniprep tool on Tue Sep 05 14:05:13 IDT 2017 from ./cx3pro_MCX314A_40g.prs
;
;; PRS  FILE FOR FALCON BENTAL 40G
;; $Id$



[PS_INFO]
Name = 4779-314A-X00_Ax
Description = ConnectX-3 Pro EN network interface card; 40/56GbE; dual-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6
PRS_name    = cx3pro_MCX314A_40g.prs

[ADAPTER]
PSID = ATT1090111023
pcie_gen2_speed_supported = true
pcie_gen3_speed_supported = true
adapter_dev_id = 0x1007
silicon_rev = 0x00

gpio_mode1 = 0x08000001
gpio_mode0 = 0x04d042fe
gpio_default_val = 0x0f287f9f
gpio_pull_up = 0xff2baf9f
gpio_pull_enable = 0xfbabbfef

receiver_detect_en = true
vdd_change_to_1_offset = 7

nv_cfg_en = true

[HCA]
hca_header_subsystem_vendor_id = 0x117c
hca_header_device_id = 0x1007
hca_header_subsystem_id = 0x0091
hca_header_class_code = 0x020000
eth_xfi_en = true
mdio_en_port1 = 0
pcie_tx_polarity = 0x00
dpdp_en = false
cmpl_timeout_default_value_en = 1

[IB]
mlpn_en_port0 = true
mlpn_en_port1 = true
phy_type_port1 = XFI
phy_type_port2 = XFI
module_power_level_supported_port0 = 5
module_power_level_supported_port1 = 5

ext_phy_board_port1 = FALCON
ext_phy_board_port2 = FALCON
gen_guids_from_mac = true
do_sense = false
ref_clk_to_use = 0
new_gpio_scheme_en = true
read_cable_params_port1_en = true
read_cable_params_port2_en = true

cx3_spec1_3_ib_support_port0 = true
cx3_spec1_3_ib_support_port1 = true
cx3_spec1_2_ib_support_port0 = true
cx3_spec1_2_ib_support_port1 = true
spec1_3_fdr14_ib_support_port0 = true
spec1_3_fdr14_ib_support_port1 = true
spec1_3_fdr10_ib_support_port0 = true
spec1_3_fdr10_ib_support_port1 = true
mellanox_ddr_ib_support        = true
mellanox_qdr_ib_support        = true

port1_802_3ap_cr4_enable = true
port2_802_3ap_cr4_enable = true
port1_802_3ap_cr4_ability = true
port2_802_3ap_cr4_ability = true
port1_802_3ap_56kr4_ability = true
port2_802_3ap_56kr4_ability = true

center_mix90phase = true

;;Logic lane to Serdes mapping
tx_logic_0_serdes = 0
tx_logic_1_serdes = 1
tx_logic_2_serdes = 2
tx_logic_3_serdes = 3
rx_logic_0_serdes = 3
rx_logic_1_serdes = 2
rx_logic_2_serdes = 1
rx_logic_3_serdes = 0

tx_logic_4_serdes = 4
tx_logic_5_serdes = 5
tx_logic_6_serdes = 6
tx_logic_7_serdes = 7
rx_logic_4_serdes = 7
rx_logic_5_serdes = 6
rx_logic_6_serdes = 5
rx_logic_7_serdes = 4

eth_tx_lane_polarity_port1 = 0xf
eth_tx_lane_polarity_port2 = 0xf
eth_rx_lane_polarity_port1 = 0x0
eth_rx_lane_polarity_port2 = 0xf
tx_lane_polarity_port1 = 0xf
tx_lane_polarity_port2 = 0xf

; start of '#include "include_QSFP_serdes_prams_bental.h"'

;;Serdes parameters
port0_nego_fdr_mask_en = 0xfffc
port1_nego_fdr_mask_en = 0xfffc
port0_nego_fdr10_mask_en = 0xfffc
port1_nego_fdr10_mask_en = 0xfffc

nego_rx4_slicer_ind_en = 255
nego_rx4_slicer1_enable = 8
nego_rx4_slicer2_enable = 8
nego_rx4_ffe_tap0 = 94
nego_rx4_ffe_tap1 = 134
nego_rx4_ffe_tap2 = 245
nego_rx4_ffe_tap3 = 135
nego_rx4_ffe_tap4 = 171

nego_rx9_ffe_tap0=84
nego_rx9_ffe_tap1=164
nego_rx9_ffe_tap2=251
nego_rx9_ffe_tap3=132
nego_rx9_ffe_tap4=140

nego_rx15_ffe_tap3 = 140
nego_rx15_ffe_tap1 = 140

nego_rx10_ffe_tap3 = 140
nego_rx10_ffe_tap1 = 140

nego_rx8_ffe_tap3 = 140
nego_rx8_ffe_tap1 = 140

force_rx0_slicer_ind_en = 0x0
force_rx0_slicer1_enable = 0x0
force_rx0_slicer2_enable = 0x0
force_rx0_ffe_tap0 = 0xff
force_rx0_ffe_tap1 = 0x80
force_rx0_ffe_tap2 = 0x80
force_rx0_ffe_tap3 = 0x80
force_rx0_ffe_tap4 = 0x80

force_tx0_ob_preemp_pre = 0x40
force_tx0_ob_preemp_post = 0x0
force_tx0_ob_preemp_main = 0x7f
force_tx0_preemp = 0x0
force_tx0_pre_polarity = 0x1
force_tx0_post_polarity = 0x1
force_tx0_main_polarity = 0x0

force_rx2_slicer_ind_en = 0xeb
force_rx2_slicer1_enable = 0x0
force_rx2_slicer2_enable = 0x0
force_rx2_ffe_tap0 = 0x64
force_rx2_ffe_tap1 = 0x80
force_rx2_ffe_tap2 = 0xde
force_rx2_ffe_tap3 = 0x80
force_rx2_ffe_tap4 = 0x46

force_tx2_ob_preemp_pre = 0x30
force_tx2_ob_preemp_post = 0x0
force_tx2_ob_preemp_main = 0x7f
force_tx2_preemp = 0x0
force_tx2_pre_polarity = 0x1
force_tx2_post_polarity = 0x1
force_tx2_main_polarity = 0x0

force_rx3_slicer_ind_en = 0xff
force_rx3_slicer1_enable = 0x8
force_rx3_slicer2_enable = 0x8
force_rx3_ffe_tap0 = 0x6c
force_rx3_ffe_tap1 = 0x80
force_rx3_ffe_tap2 = 0xff
force_rx3_ffe_tap3 = 0x80
force_rx3_ffe_tap4 = 0x80

force_tx3_ob_preemp_pre = 0xc
force_tx3_ob_preemp_post = 0x7f
force_tx3_ob_preemp_main = 0x45
force_tx3_preemp = 0x0
force_tx3_pre_polarity = 0x1
force_tx3_post_polarity = 0x0
force_tx3_main_polarity = 0x1
force_tx3_ob_bias = 0xa

auto_ddr_tx_options = 2
auto_ddr_rx_options = 1

auto_qdr_tx_options = 6
auto_qdr_rx_options = 7

preset_tx_fdr_set12_ob_preemp_pre = 17
preset_tx_fdr_set12_ob_preemp_post = 0
preset_tx_fdr_set12_ob_preemp_main=25
preset_tx_fdr_set12_preemp = 0
preset_tx_fdr_set12_pre_polarity = 1
preset_tx_fdr_set12_post_polarity = 1
preset_tx_fdr_set12_main_polarity = 0
preset_tx_fdr_set12_ob_bias = 5

preset_tx_fdr_set13_ob_preemp_main =40
preset_tx_fdr_set13_ob_preemp_pre = 28
preset_tx_fdr_set13_ob_preemp_post = 0
preset_tx_fdr_set13_preemp = 0
preset_tx_fdr_set13_pre_polarity = 1
preset_tx_fdr_set13_post_polarity = 1
preset_tx_fdr_set13_main_polarity = 0
preset_tx_fdr_set13_ob_bias = 5

preset_tx_fdr_set14_ob_preemp_main = 35
preset_tx_fdr_set14_ob_preemp_pre = 25
preset_tx_fdr_set14_ob_preemp_post = 0
preset_tx_fdr_set14_preemp = 0
preset_tx_fdr_set14_pre_polarity = 1
preset_tx_fdr_set14_post_polarity = 1
preset_tx_fdr_set14_main_polarity = 0
preset_tx_fdr_set14_ob_bias = 5

preset_tx_fdr_set15_ob_preemp_main = 30
preset_tx_fdr_set15_ob_preemp_pre = 20
preset_tx_fdr_set15_ob_preemp_post = 0
preset_tx_fdr_set15_preemp = 0
preset_tx_fdr_set15_pre_polarity = 1
preset_tx_fdr_set15_post_polarity = 1
preset_tx_fdr_set15_main_polarity = 0
preset_tx_fdr_set15_ob_bias = 5

preset_tx_mask = 0xfffe

aba_mask0_start = 0
aba_mask0_end   = 3
aba_mask0 = 0x1000
aba_mask1_start = 4
aba_mask1_end   = 5
aba_mask1 = 0x8000
aba_mask2_start = 6
aba_mask2_end   = 10
aba_mask2 = 0x4000
aba_mask3_start = 11
aba_mask3_end   = 16
aba_mask3 = 0x2000

; ABA 40GE
aba_tx2_ob_preemp_pre = 20
aba_tx2_ob_preemp_main = 42
aba_tx2_ob_preemp_post = 8
aba_tx2_ob_bias = 8
aba_tx2_pre_polarity = 1
aba_tx2_post_polarity = 1
aba_tx2_main_polarity = 0

;;3m
aba_tx3_ob_preemp_pre = 22
aba_tx3_ob_preemp_main = 42
aba_tx3_ob_preemp_post = 5
aba_tx3_ob_bias = 8
aba_tx3_pre_polarity = 1
aba_tx3_post_polarity = 1
aba_tx3_main_polarity = 0

aba_tx4_ob_preemp_pre = 26
aba_tx4_ob_preemp_main = 42
aba_tx4_ob_preemp_post = 3
aba_tx4_ob_bias = 8
aba_tx4_pre_polarity = 1
aba_tx4_post_polarity = 1
aba_tx4_main_polarity = 0

aba_tx5_ob_preemp_pre = 60
aba_tx5_ob_preemp_main = 90
aba_tx5_ob_preemp_post = 8
aba_tx5_ob_bias = 8
aba_tx5_pre_polarity = 1
aba_tx5_post_polarity = 1
aba_tx5_main_polarity = 0

aba_tx6_ob_preemp_pre = 80
aba_tx6_ob_preemp_main = 110
aba_tx6_ob_preemp_post = 10
aba_tx6_ob_bias = 8
aba_tx6_pre_polarity = 1
aba_tx6_post_polarity = 1
aba_tx6_main_polarity = 0

aba_tx7_ob_preemp_pre = 75
aba_tx7_ob_preemp_main = 110
aba_tx7_ob_preemp_post = 15
aba_tx7_ob_bias = 8
aba_tx7_pre_polarity = 1
aba_tx7_post_polarity = 1
aba_tx7_main_polarity = 0

aba_fdr_tx16_ob_preemp_pre = 17
aba_fdr_tx16_ob_preemp_post = 0
aba_fdr_tx16_ob_preemp_main=25
aba_fdr_tx16_preemp = 0
aba_fdr_tx16_pre_polarity = 1
aba_fdr_tx16_post_polarity = 1
aba_fdr_tx16_main_polarity = 0
aba_fdr_tx16_ob_bias = 5

aba_fdr_tx17_ob_preemp_main =46
aba_fdr_tx17_ob_preemp_pre = 32
aba_fdr_tx17_ob_preemp_post = 0
aba_fdr_tx17_preemp = 0
aba_fdr_tx17_pre_polarity = 1
aba_fdr_tx17_post_polarity = 1
aba_fdr_tx17_main_polarity = 0
aba_fdr_tx17_ob_bias = 3

aba_fdr_tx18_ob_preemp_main = 50
aba_fdr_tx18_ob_preemp_pre = 32
aba_fdr_tx18_ob_preemp_post = 0
aba_fdr_tx18_preemp = 0
aba_fdr_tx18_pre_polarity = 1
aba_fdr_tx18_post_polarity = 1
aba_fdr_tx18_main_polarity = 0
aba_fdr_tx18_ob_bias = 3

aba_fdr_tx19_ob_preemp_main = 60
aba_fdr_tx19_ob_preemp_pre = 30
aba_fdr_tx19_ob_preemp_post = 0
aba_fdr_tx19_preemp = 0
aba_fdr_tx19_pre_polarity = 1
aba_fdr_tx19_post_polarity = 1
aba_fdr_tx19_main_polarity = 0
aba_fdr_tx19_ob_bias = 3

aba_index0_start = 0
aba_index0_end   = 3
aba_index0 = 0
aba_index1_start = 4
aba_index1_end   = 5
aba_index1 = 3
aba_index2_start = 6
aba_index2_end   = 9
aba_index2 = 2
aba_index3_start = 10
aba_index3_end   = 16
aba_index3 = 1

aba_rx2_slicer_ind_en = 0xeb
aba_rx2_slicer1_enable = 0x0
aba_rx2_slicer2_enable = 0x0
aba_rx2_ffe_tap0 = 0x80
aba_rx2_ffe_tap1 = 0x68
aba_rx2_ffe_tap2 = 0xd7
aba_rx2_ffe_tap3 = 0x80
aba_rx2_ffe_tap4 = 0x5a

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;; SFP+ section. all QSFP can be converted to SFP+ using QSA adapter.;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; ETH connected to third party device
aba_non_mlpn_tx8_ob_preemp_pre = 5
aba_non_mlpn_tx8_ob_preemp_post = 0
aba_non_mlpn_tx8_ob_preemp_main = 65
aba_non_mlpn_tx8_ob_bias = 8
aba_non_mlpn_tx8_pre_polarity = 1
aba_non_mlpn_tx8_post_polarity = 1
aba_non_mlpn_tx8_main_polarity = 0
aba_non_mlpn_tx8_preemp = 0


nego_eth_rx12_slicer_ind_en = 0xff
nego_eth_rx12_slicer1_enable= 0x8
nego_eth_rx12_slicer2_enable= 0x8
nego_eth_rx12_ffe_tap0=241
nego_eth_rx12_ffe_tap1=128
nego_eth_rx12_ffe_tap2=61
nego_eth_rx12_ffe_tap3=99
nego_eth_rx12_ffe_tap4=128

; end of '#include "include_QSFP_serdes_prams_bental.h"'

[PLL]
lbist_en  = 0
lbist_shift_freq  = 3
flash_div = 0x3
lbist_array_bypass = 1
lbist_pat_cnt_lsb = 0x2
core_f = 60
core_r = 14
core_od = 2
en_427_mhz = true

[FW]
flash_has_suspend_resume = 0
log_flashdev_size = 21
log_flash_sector_size = 6

 

[nbritton]

I sorta got it working with the regular ConnectX-3 MCX353A-FCBT, all I did was generate a new firmware image using the [HCA] section from the INI file above and also changed the PSID. My device is a single port, so everything below relates to the single port ATTO FastFrame NQ41, steps for the dual port could be slightly different.

These are the commands / steps I did:

# Backup current firmware on the Mellanox card and the INI file:
flint -d /dev/mst/mt4099_pci_cr0 ri DEL1100001019.bin;
flint -d /dev/mst/mt4099_pci_cr0 dc > DEL1100001019.ini;

# Create a new INI file using the INI file from the previous step as the foundation:
cp -a DEL1100001019.ini ATT1060111023.ini;

# Swap out the [HCA] section and change PSID under the [ADAPTER] section with the following:
vim ATT1060111023.ini

[ADAPTER]
PSID = ATT1090111023

[HCA]
hca_header_subsystem_vendor_id = 0x117c
hca_header_device_id = 0x1007
hca_header_subsystem_id = 0x0091
hca_header_class_code = 0x020000
eth_xfi_en = true
mdio_en_port1 = 0
pcie_tx_polarity = 0x00
dpdp_en = false
cmpl_timeout_default_value_en = 1

# Download Mellanox's firmware development kit, you need the fw-ConnectX3-rel.mlx file.
# I can't find "fw-ConnectX3-rel.mlx" version 2.42.5000, if you have a copy send it my way please.
wget http://content.mellanox.com/firmware/ConnectX3-rel-2_40_5030.tgz;
tar -axf ConnectX3-rel-2_40_5030.tgz;

# Download Mellanox's Firmware Tools package and install, and also install the extra mft-oem package, mft-oem contains the tools necessary to create new firmware images.
wget http://content.mellanox.com/MFT/mft-4.14.0-105-x86_64-deb.tgz;
tar -axf mft-4.14.0-105-x86_64-deb.tgz;
cd mft-4.14.0-105-x86_64-deb;
./install.sh;
dpkg -I ./DEBS/mft-oem_4.14.0-105_amd64.deb;

# Create a new firmware image using the fw-ConnectX3-rel.mlx and ATT1060111023.ini files, output file is fw-ConnectX3-rel-2_40_5030-ATT1060111023.bin:
mlxburn -wrimage fw-ConnectX3-rel-2_40_5030-ATT1060111023.bin -fw fw-ConnectX3-rel.mlx -conf ATT1060111023.ini;

# Start mst services if not already started:
mst start;

# Flash new firmware image onto the Mellanox card:
flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_40_5030-ATT1060111023.bin burn;

# Reboot for the new firmware image to take effect:
reboot;

After rebooting the Linux host, the Subsystem now identifies as ATTO, vfio-pci is attached because I'm running macOS Mojave inside of a KVM VM on a Ubuntu 20.04 system, so I'm doing PCIe device passthrough of the Mellanox card into the VM... this could potentially be the cause for why it's not working right... I still need to get a 10 meter QSFP cable to reach all the way to my Mac Pro 2012.

root@r920-cmwhv52:~/Mellanox# lspci -vs 41:00
41:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
    Subsystem: ATTO Technology, Inc. FastFrame NQ41
    Flags: bus master, fast devsel, latency 0, IRQ 139, NUMA node 2
    Memory at d3d00000 (64-bit, non-prefetchable) [disabled] [size=1M]
    Memory at d3000000 (64-bit, prefetchable) [disabled] [size=8M]
    Expansion ROM at d3e00000 [disabled] [size=1M]
    Capabilities: [40] Power Management version 3
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable- Count=128 Masked-
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [c0] Vendor Specific Information: Len=18 <?>
    Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [148] Device Serial Number 50-6b-4b-03-00-4c-fa-c0
    Capabilities: [154] Advanced Error Reporting
    Capabilities: [18c] Secondary PCI Express
    Kernel driver in use: vfio-pci

From within the macOS Mojave VM I get the following:

Stumbleine:FastFrame2 root# ./atinfo -c 1 -i all

######################################################################
Channel 1: ATTO FastFrame NQ41
######################################################################

Node Address:
Permanent MAC Address:   00:00:00:00:00:00
Current MAC Address:     00:00:00:00:00:00
PCI Vendor ID:           15B3
PCI Device ID:           1007
PCI Subsystem Vendor ID: 117C
PCI Subsystem Device ID: 0090
PCI Location:            1.0.0
PCI Class Code:          020000
PCI Revision ID:         0
Current PCI Link Width:  x8
Maximum PCI Link Width:  x8
Current PCI Link Speed:  8 GT/s (Gen3)
Maximum PCI Link Speed:  8 GT/s (Gen3)
Interrupt Mode:          Legacy
Driver Name:             ATTOFastFrame2.kext
Driver Version:          1.11.0f1
Firmware Version:        2.40.5030
Controller State:        Degraded

 

[nbritton]

I found a Mellanox QSFP to SFP+ adapter so I was able to try this in my Mac Pro 2012, unfortunately it did not work... but it feels very close to working... I'm willing to bet if I had a genuine ConnectX-3 Pro (I only have regular CX3 right now, not the Pro version) it would work. In the screenshot below it appears the card lost it's MAC addresses. I checked with flint and it's still there, but on closer investigation it appears the ATTO driver can't access the NVRAM region of the regular non-pro CX3 card. I'm going to stop here until I can get my hands on a real CX3 Pro.

 

[simonjii]

Hi. This is extremely interesting. How sure are you that it'll work with the ConnectX-3 Pro? I'm about to get one on Ebay to test that with our 2013 MacPro (in external TB PCI box) and our server.
Do you think those specs should be good? Model No: CX314A, P/N: MCX314A-BCCT, Pro version, CONNECTX-3Pro EN 40 Gigabit Ethernet Adapter PCI-E 2-Port MELLANOX.

 

[nbritton]

Hi. This is extremely interesting. How sure are you that it'll work with the ConnectX-3 Pro? I'm about to get one on Ebay to test that with our 2013 MacPro (in external TB PCI box) and our server.
Do you think those specs should be good? Model No: CX314A, P/N: MCX314A-BCCT, Pro version, CONNECTX-3Pro EN 40 Gigabit Ethernet Adapter PCI-E 2-Port MELLANOX.

Yes I would be willing to bet that it'll work, and the MCX314A-BCCT card has the correct device id of 4103. The most important thing to ensure is that the device id matches the one in the ATTO card, and this one does. Best of luck, please post your results... unfortunately I haven't had the spare cash to buy one of these cards to test yet due to the whole covid-19 pandemic. Your worst case is the driver simply won't work, and if that happens then you can flash it back to the stock firmware and just return it to the seller on eBay.

 

[evilzardoz]

Hey,

This is really cool! Shame all my Mellanox Connect-X 3 cards are either the CX354A or CX353A. Would be curious if there was a way to get those to work!

 

[nbritton]

Hey,

This is really cool! Shame all my Mellanox Connect-X 3 cards are either the CX354A or CX353A. Would be curious if there was a way to get those to work!

There is probably a way to hack the ATTO driver to use stock Mellanox cards but I don’t know how to do that off the top of my head and I don’t have any slack time to research it right now.

 

[simonjii]

Hi. I just ordered the MCX314A-BCCT and should have it by 1st of June at the latest. I'll report here or ask questions.

 

[simonjii]

Hi nbritton

I got my Connect X3-pro and flashed the firmware in Linux. Put it in my Mac Pro.
But unfortunately it looks like I'm at the exact same point like you were:

I did manage to flash the firmware again on the mac but it doesn't change anything. Looks like the same issue with NVRAM. Do you have any ideas on how to proceed?

Edit: I got a bit further. I could restore the NVRAM to default settings with atinicnvr and now the NVRAM not found message disappeared. Still the same in the ATTO config tool. Maybe we would need a proper ATTO Nvram file to copy onto the card?

 

[Eric's MacRumors Account]

Hi all,

This is some super interesting stuff, any updates at the moment?

Cheers!

 

[simonjii]

Hi
I didn't manage to get it to work. In the end I got a Chelsio 40gbe pci card, which are sold as cheap as the Mellanox on Ebay and works flawlessly with an official driver in OS X

 

[DCswitch]

Hi
I didn't manage to get it to work. In the end I got a Chelsio 40gbe pci card, which are sold as cheap as the Mellanox on Ebay and works flawlessly with an official driver in OS X

Hi Simon, I'm going down this rabbit hole now. I am having trouble with the Chelsio T62100-LP-CR. I have a connection to my TrueNAS and it works great, but then it always drops the shares. I just ordered a used T5 40G to see how it goes.

Can you tell me which version of macOS you're running and any special tricks you may have done to get 'er running smoothly?

 

[evilzardoz]

Has anyone been successful? Otherwise, what Chelsio cards do people recommend and have they been stable? I keep getting weird drops during large transfers on a Solarflare card and wouldn't mind better performance. Drivers are 10.9 vintage, so not surprised I'm running into issues.

Do the Chelsio cards work with QSFP to SFP+ adapters? Generic QSFP or branded?

What about device support in Big Sur? The other elephant in the room is obviously the Apple Silicon situation...

 

[DCswitch]

Has anyone been successful? Otherwise, what Chelsio cards do people recommend and have they been stable? I keep getting weird drops during large transfers on a Solarflare card and wouldn't mind better performance. Drivers are 10.9 vintage, so not surprised I'm running into issues.

Do the Chelsio cards work with QSFP to SFP+ adapters? Generic QSFP or branded?

What about device support in Big Sur? The other elephant in the room is obviously the Apple Silicon situation...

I've been using the Chelsio T580-LP-CR for a few days and I can say it definitely did not drop shares in a Thunderbolt enclosure. I just installed into my Mac Pro 7,1 and so far it's running fine. It's only been 10 minutes though, so I'll report back if it drops shares. The 100G card kept dropping shares in the Mac Pro, but I was having better luck in the TB enclosure.

I'm not sure about the SFP+, but I would just buy two so you can run 40G. Chelsio uses generic/Cisco.

 

[churek]

Hi Simonji,
I have the original atto card. May I help you with something ?

 

[willowthewisp]

Dear Gentlemans,
did you read the log files after flashing ?
The cross flashed cards are obviously missing their MAC addresses !
Without the MAC the Mellanox cards are known to disable the driver in Windows, Linux and FreeBSD - macOS won’t make an exception.
Who will be the first to flash the MAC back ? (flint and mst can do that).
Don’t worry - you can’t brick the Mellanox cards - there is a jumper labeled FNP - firmware not present.
If you connect these two pins the card will be visible to the os but the firmware (if corrupted) will not be loaded (livefish-mode).
Also you can access and flash the firmware via the onboard I2C header (SDA/SCX).
The NVRAM is stored in an 24C32 i2c device - to my knowledge it’s not used by macOS.

 

[TrumanLA]

Anyone notice ATTO FastFrame NQ41 / NQ42 are rebranded ConnectX-3 CX353A / CX354A cards?
The FW .bin is on ATTO's website & may only require "Cross-Flashing" a generic CX3's FW with ATTO's.
Which should be doable using Mellanox's mstflint + the Mellanox Firmware Tool package.

I paraphrased your first entry ... which was also referenced in another MR discussion re TB + SFP28

Are there no adapters which are natively supported with MacOS ..?

Has anyone been successful? Otherwise, what Chelsio cards do people recommend and have they been stable? I keep getting weird drops during large transfers on a Solarflare card and wouldn't mind better performance. Drivers are 10.9 vintage, so not surprised I'm running into issues.

Do the Chelsio cards work with QSFP to SFP+ adapters? Generic QSFP or branded?

What about device support in Big Sur? The other elephant in the room is obviously the Apple Silicon situation...

Exactly:

It's all fine n dandy to be committed to legacy products (Chelsio is even provided drivers for Catalina).
BUT ... once you want to use an M1 or newer ... or Monterey ... the plot thickens.

On the other hand -- there is actually a project somewhere to try to create a unified kernel of FreeBSD and MacOS.
Though ... sounds like a poor idea to hold one's breath waiting for such a lofty endeavor to come to fruition. lol.

Good read here ... I will note: The pictures of ATTO's FF N352 and N312 (the latter being quite tenuous as nothing but a 2019 Mac Pro presently has the bandwidth to take advantage of it -- as TB3 is limited to 22Gb/s DATA within MacOS and only in Linux can one tune the reserved bandwidth in a way that's beyond my ability) ... but still, the N351 / N352 look to me..? Like a ConnectX-4. Specifically, the Mellanox MCX416A ... as in, every single chip appears to be identical per pictures I've found of each.

I'm not saying I'm correct; but it just looks that way to me.

Anyway, I have an older ThunderLink (TB1 or 2 to SFP+) ... which I'm tempted to open up and see if it'll recognize any other adapters if there's a PCIe slot in it. If anything good comes of it I'll report back.

Otherwise -- good luck.

 

[reasonsandreasons]

Who will be the first to flash the MAC back ? (flint and mst can do that).

I'd be really interested to see what effect this has. Looking at the man page, it looks like it might be as easy as appending --mac XX:XX:XX:XX:XX:XX (where XX:XX:XX:XX:XX:XX is an arbitrary MAC) after the burn command to flash the card. Especially worth trying if the MAC is also blank in Linux.

I might track one of these down myself. An easily-available 40 gigabit ethernet card backed by drivers from a company as responsive as ATTO would be really great, especially as DriverKit looms.

 

[thefloyd]

So I’ve got a cx3pro crossflashed to the ATTO firmware but yeah, same issue here. The card works fine in the Linux host where I flashed it - MAC addresses show up, interface comes up and is usable, etc but putting it in a mac results in the dreaded 00’d MAC address. I haven’t poked around but I’d guess it’s something silly like the NVRAM being at a different i2c address on the card since the atto tools specifically say they can’t talk to the nvram (based on other posts above, haven’t personally tried this yet).

 

[thefloyd]

Dear Gentlemans,
did you read the log files after flashing ?
The cross flashed cards are obviously missing their MAC addresses !
Without the MAC the Mellanox cards are known to disable the driver in Windows, Linux and FreeBSD - macOS won’t make an exception.
Who will be the first to flash the MAC back ? (flint and mst can do that).
Don’t worry - you can’t brick the Mellanox cards - there is a jumper labeled FNP - firmware not present.
If you connect these two pins the card will be visible to the os but the firmware (if corrupted) will not be loaded (livefish-mode).
Also you can access and flash the firmware via the onboard I2C header (SDA/SCX).
The NVRAM is stored in an 24C32 i2c device - to my knowledge it’s not used by macOS.

It’s definitely not “missing” MAC addresses because the card boots up fine and interacts with the local network as expected after the crossflash on a Linux host. I suspect something silly like a changed i2c address for the NVRAM on the ATTO card since this is super simple to change in a mass-produced product and makes it “just” incompatible enough for people who simply cross-flash.

 

[thefloyd]

Sadly whatever is different can't be explained by just the .ini files. A quick diff between my card flashed to stock cx3pro firmware versus the atto firmware yields nothing too interesting:

root@thefloyd-dt:~/atto# diff atto.ini stock.ini
9,10c9,10
< Name = 4779-314A-X00_Ax
< Description = ConnectX-3 Pro EN network interface card; 40/56GbE; dual-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6
---
> Name = MCX314A-BCC_Ax
> Description = ConnectX-3 Pro EN network interface card; 40GigE; dual-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6
14c14
< PSID = ATT1090111023
---
> PSID = MT_1090111023
32d31
< hca_header_subsystem_vendor_id = 0x117c
34c33
< hca_header_subsystem_id = 0x0091
---
> hca_header_subsystem_id = 0x0006
root@thefloyd-dt:~/atto#

I think we're going to need a dump of the nvram from someone with the ATTO card. Anyone here able to do that?

 

[thefloyd]

Hi Simonji,
I have the original atto card. May I help you with something ?

hi churek:

A copy of the NVRam contents would be ideal. On my machine with the current drivers it looks like the command is (from a terminal window):

cd /Applications/ATTO/FastFrame2

sudo ./atnetnvr2 -s ~/Documents/attonvram.backup

(alternatively, sudo ./atnetnvr2 -s /full/path/to/attonvram.backup if you don't want it in your personal Documents folder)

You should end up with a file called attonvram.backup in your 'Documents' folder (modern OSX may prompt you to allow things to write to 'Documents' when you execute this, feel free to save anywhere you're comfortable, this just seemed easy). It doesn't appear from the command line help that you need to specify a device specifically:

thefloyd@Patricks-MacBook-Air-2 FastFrame2 % ./atnetnvr2

Usage:

atnetnvr2 <options> [command]

Options:

-a {length} Set the TX ring buffer length.
-b {length} Set the RX ring buffer length.
{length} is the number of ring buffer entries.
Valid values are 64-8192. 0 indicates driver default.
-c {channel} Selects a specific controller channel for the operation,
starts at 1. For dual port adapters, you must also
specify the port number with -c {channel}:{port},
eg. -c 1:1 for the first port and -c 1:2 for the second.
-d {feature} Disable an NVRAM feature.
-e {feature} Enable an NVRAM feature.
{feature} values:
adapcoal Adaptive interrupt coalescing.
checksum Hardware checksum offloads.
dcb Enable DCB.
rsc Receive Side Coalescing.
tso Enable Transmit Segmentation Offload.
-h Display extended help.
-i {usecs} Set the interrupt coalescing. {usecs} is a number of
microseconds from 0 to 1000.
-l List the controllers in the system.
-m {bytes} Threshold size for inline transmits.
Valid values are 0-104, with 0 disabling inlining.
-p Print the contents of the NVRAM when other operations
are complete.
-q {length} TX queue length.
Valid values are 8-8192. 0 indicates driver default.
-r {filename} Restore the NVRAM from a file.
-s {filename} Save the NVRAM to a file.
-t Restore the NVRAM to default settings.
-v Print tool version information.
-z {length} Set the Transmit Segmentation Offload length.
{length} is the number of TSO bytes.
Valid values are 1500-65536. 0 indicates driver default.


No options found.


I'm going to try and poke at this more at some point today myself, but without a 'genuine' NVRam we might not get far.