How to reason about the 4 TB3 ports?

[wagle]

I'd like to be able to network my two (2) mac minis at the "up to 40Gb/s". How do I do that?

Reality is confusing:

(1) I can get 21GbE by directly pairing either port 2 with port 2 OR port 4 with port 4.

(2) I can get (only about) 17GbE by directly pairing either port 1 with port 1 OR port 3 with port 3.

(3) IIRC if I pair ports 2 and 4 on one with the corresponding ports on the other, I can get 33GbE.

(4) Most other double pairings get 17GbE.

(5) Using port 1 seemed to disable port 2 from at least networking. Likewise 3 and 4.

---

I think I don't have the above right,

What IS right? How do I reason about how to configure TB3 "40GbE" networking? What the heck is that bus doing here? Is there documentation?

Thanks!

 

[djc6]

Are you doing this via the "thunderbolt bridge" network interface?

 

[wagle]

Yes, that's an important point, sorry. Unbridged 2-2 (or 4-4) gets 21GbE, but bridged single only 15-17GBE (I think). Bridged double 2-2 and 4-4 got 33GbE.

Sorry, can't verify anything at the moment since I had to return the extra cable within 14 days, and it's holiday season, so I thought I'd wait for responses and things to try before giving experimentation another go.

 

[djc6]

Can you tell me a little more about this setup, especially software you are using for your performance tests? I'm not sure what you mean by 'unbridged'.

What application are you using to test the speed of this connection? If you are seeing 33Gb/s between two macs with TCP/IP that sounds pretty amazing already. I'd need to know more about your tests - to really saturate the connection you may need to do some kernel level tcp tuning that I wouldn't know how to do on macos - just thinking of my similar experiments on Linux.
[doublepost=1545672412][/doublepost]I was kind of curious about this. I ran a thunderbolt 3 cable between mac mini 2018 and my mid-2017 macbook pro.

I ran iperf3 client on mac mini, iperf3 server on the macbook pro. I was able to get 13Gb/s between the macs - but I didn't do much tuning. I tried increasing number of parallel streams with (-P) option and also running multiple instances of iperf3 server/client simultaneously.

 

[wagle]

I used iperf3. My attempts to tune TCP degraded things.

"Unbridged" is just the network settings for an individual port. "Bridged" is grouping them together with that app.

You have to use (something like) an active Thunderbolt cable and "compatible" ports on both ends of a single cable to get above 15GbE or so.

I think its the OS and the hardware that are playing funny tricks, not the cable or iperf3 or the tcp stack, especially since ports 1 and 3 do different things than ports 2 and 4.

I just want to know how to configure the network hardware correctly (so I can blame iperf3 and or TCP tuning), which I don't think I am doing.

 

[djc6]

I think that is piece of puzzle I was missing. I didn't realize you were using multiple cables between both machines simultaneously. I'm using thunderbolt bridge interface with a single cable.

I used a Belkin half meter passive thunderbolt cable allegedly rated at 40Gb/s:

https://www.belkin.com/us/p/P-F2CD084/

Here is a discussion about how much bandwidth is actually available via the pair of thunderbolt controllers, especially if one specs the machine with a 10Gb Ethernet port:

https://forums.macrumors.com/threads/2018-mac-mini-pcie-lane-tb3-discussion-lounge.2150992/

 

[wagle]

I tried a short enough passive cable, and its performance (single) was "miserable" (15GbE or so?). so I returned it.

The mysterious (active? passive? who knows?) Apple brand TB3-TB3 cables worked as well and even more expensive (but longer active cables).

I thought I'd read that thread, but there seems to be more now.

I do also have the 10GbE ports.
[doublepost=1545683356][/doublepost]Oh yeah, one important question is "why does having networking enabled on port 1 disable port 2?".