Resolved (7,1) Can't Achieve 10Gig Speeds

[jasonmvp]

Hey folks -

Has anyone else noticed your 7,1's 10Gig interfaces slowing down significantly? I'm not sure what changed and when, but I'm suspecting it was the installation of Monterey. Either way, I can barely get 1.5Gbits/sec out of either of my 10Gig interfaces when before I was able to push them nearly to 10Gbits/sec each.

A very rough diagram of the network at home. Each of the connections is a 10Gig link, either via fiber (switch to switch), DAC (server and router in basement), or Cat7 copper (Mac and PC). Each shows 10G on the corresponding switch. Using iperf3 as my test:

• Mac to Server: 1.5Gbits/sec
• Mac to router: 1.5Gbits/sec
• Mac to PC: 1.5Gbits/sec
• PC to server: ~9Gbits/sec
• PC to router: ~9Gbits/sec
• router to server: ~10Gbits/sec

Clearly something's up with the Mac.

Has anyone else noticed this? Thanks.

 

[awkward_eagle]

The default settings don't allow for full 10g over SMB. Disable packet signing using the nsmb.conf file then unmount and remount the volume.

https://support.apple.com/en-us/HT205926

If you're using a network volume that should be fast enough for Final Cut Pro libraries, it requires the extra step of enabling macOS's Server Performance Mode. Might require temporary disabling of SIP to modify.

https://support.apple.com/en-us/HT202528

This older thread covers the whole saga with Catalina and FCP but the same still applies in Monterey.

https://forums.macrumors.com/thread...-cut-pro.2224843/?post=28255627#post-28255627

Edit : Forgot to mention interface settings. Be sure the port is manually set to 10g full duplex with jumbo frames. If it's a managed switch make sure the ports on the switch match.

 

[jasonmvp]

The default settings don't allow for full 10g over SMB. Disable packet signing using the nsmb.conf file then unmount and remount the volume.

I apologize if I wasn't clear in the OP and that is my fault. Prior to Monterey, or prior to something else that I have yet to determine(!), I was getting damned near 10Gbps out of those 10GigE ports. This was verified using iperf3, not SMB or any other means.

 

[awkward_eagle]

I apologize if I wasn't clear in the OP and that is my fault. Prior to Monterey, or prior to something else that I have yet to determine(!), I was getting damned near 10Gbps out of those 10GigE ports. This was verified using iperf3, not SMB or any other means.

Just double checked the SMB connection between two 2019 MPs and its hitting full 10g. Missed the iperf3 part. Sorry for the confusion.

 

[jasonmvp]

Just double checked the SMB connection between two 2019 MPs and its hitting full 10g. Missed the iperf3 part. Sorry for the confusion.

It's OK. I'm a network architect by trade. ;-)

 

[Ludacrisvp]

Disable packet signing using the nsmb.conf file then unmount and remount the volume.
https://support.apple.com/en-us/HT205926

If you're using a network volume that should be fast enough for Final Cut Pro libraries, it requires the extra step of enabling macOS's Server Performance Mode. Might require temporary disabling of SIP to modify.
https://support.apple.com/en-us/HT202528

My reading of the man page for nsmb.conf would suggest that in modern macOS (Big Sur here) that this has no effect as this value is defaulted to no already. In fact the KB calls out this only applied to 10.3.3 and older.

Possible keywords may include:

Keyword             Section    Default    Comment
                    A B C      Values
...
signing_required    + - -      no         Turn on smb client signing
...

Any idea what Performance mode really does? and wouldn't that really need to be done on the 'server side' rather than the client here? I'm expecting (or rather assuming) that Jason's server is not a Mac.

Back to the topic at hand ...
While I don't have a 2019 Mac Pro, if I have flow control turned off on the Mac I see speeds that are terrible like you mention via iperf, where basically a 1Gb link would almost outperform the 10Gb link, with flow control enabled all things are better.

for some details on what i've got set up .. I'm using myricom 10Gb with SR fiber with finisar SFPs, hitting a mikrotik 4 port 10Gb switch and a FreeBSD 13 NAS with chelsio T3 (N320) and have also tried another myricom card which is currently installed, as my hardware is older I tend to get between 7Gbps and 7.5Gbps in each direction. I believe if I bypass the switch and direct connect I was getting 9.xGbps in iperf.

Long story short, can you check flow control settings on the mac and on each switch in the path?
Perhaps the upgrade to Monterey flipped that setting or it is no longer automatically deciding to enable flow control.

 

[jasonmvp]

Long story short, can you check flow control settings on the mac and on each switch in the path?
Perhaps the upgrade to Monterey flipped that setting or it is no longer automatically deciding to enable flow control.

Good idea, but it wasn't flow control. It was software related, however, but not Monterey specifically. Last year, I'd let the company I work for pwnz0r my Mac Pro, so I could use it for work and play. This seemed to go perfectly well, but some software push from them seems to have gimped the NICs. I'm not sure if it's JAMF related (everyone's favorite!) or the VPN software even when it's disconnected, or something else. But that was the culprit.

I figured this out with the help of the Apple tech support folks. The dude suggested rebooting in Safe mode, which I did. And voila: nearly 10Gbits/sec throughput. Dammit.

I've since de-worked my Mac since I have a corporate Macbook Pro anyhow, and am once again cranking 10Gig:

$ iperf3 -c 192.168.10.9 -P 4
Connecting to host 192.168.10.9, port 5201
[  4] local 192.168.10.10 port 50062 connected to 192.168.10.9 port 5201
[  6] local 192.168.10.10 port 50063 connected to 192.168.10.9 port 5201
[  8] local 192.168.10.10 port 50064 connected to 192.168.10.9 port 5201
[ 10] local 192.168.10.10 port 50065 connected to 192.168.10.9 port 5201

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[SUM]   0.00-10.00  sec  11.0 GBytes  9.47 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  11.0 GBytes  9.46 Gbits/sec                  receiver