Thunderbolt questions

[macman4789]

Hi,

Quick question which I'm sure someone clever can answer. The M1 Max Studio has 4 Thunderbolt ports on the back and two USB-C ports on the front.

My question is about the throughput of all these ports, my very limited understanding is that it depends on the number of bus? Which would determine the maximum throughput utilised at once? So for the older Intel 16" MacBook Pro's I think there was one bus for either side of the machine so you could in theory have 80Gbps at one time? 2 x 40Gbps however if you used two ports on the same side that would be halved? So roughly 20Gbps each?

If that is correct, my question is what is the situation with the new M1 Max Studio which has 4 TB ports at the back and 2 USB-C ports at the front, if I used all 4 TB ports at once how much data transfer would I get at once with the M1 Max? Is it different with these?

Additionally, if I was using the TB ports, would that affect the 2 x 10Gbps USB-C ports at the front? The 10Gig Ethernet port?

Just trying to understand how it works.

Many thanks in advance.

 

[nosnhojm]

These are TB4 ports, so each has their own 40Gbps bus. Not sure on the USB-C ports, but I'm guessing the two USB-C, SDXC reader, and ethernet share a single thunderbolt lane (40Gbps total).

 

[joevt]

The speed per Thunderbolt port is ≈22-25 Gbps ≈ 2700 MB/s
The speed per USB port is < 9.7 Gbps ≈ 1000 MB/s

On an Intel Mac with two discrete Thunderbolt 3 controllers, you can get ≈22 Gbps per port, ≈23 Gbps per Thunderbolt controller, ≈41 Gbps total (two Thunderbolt controllers).

On an Intel Mac with integrated Thunderbolt 3 controllers on the Ice Lake CPU, you can get ≈22 Gbps per port, up to ≈38 Gbps for any combination of ports.

On an M1 Mac where everything is integrated (except the extra USB-C non-Thunderbolt ports of the iMac), each port is ≈23 Gbps as usual, but no-one has tested the max total. You need 6 drives for the Studio or less if the max is reached before getting to 6. For example, the maximum for Intel Macs is achieved with just 3 drives.

Find benchmarks posted elsewhere for example numbers for individual drives.

The easiest way to test total bandwidth is to connect USB NVMe and Thunderbolt NVMe drives that are known to outperform each port. Then measure total bandwidth using ATTO Disk Benchmark. It's not been updated for Apple Silicon but I think it should work. The useful feature it has over other benchmarks is that it can do multiple drives at once without needing to create a RAID 0 (which you can't do with a network drive) and it does various I/O sizes so you can see how increasing the I/O size improves performance (maybe skip lower I/O sizes to make the benchmark faster). I measured my Gigabit Ethernet at ≈116.75 MB/s max, which is 0.934 Gbps.

I haven't seen an ioreg output for either Studio. I believe the one with 6 Thunderbolt ports probably just has 6 separate Thunderbolt devices so the limit is whatever the CPU can do. The Studio with 2 USB-C ports might use a discrete USB controller. If it's the same one as the M1 iMac, then it may allow 15 Gbps total?

 

[macman4789]

Thanks for you replies guys.

So in theory with the M1 Max each port would give 40Gbps at the same time so could have 160Gbps output all at once?

So does each port have its own bus or is it different for Apple Silicon?

Thanks

 

[joevt]

So in theory with the M1 Max each port would give 40Gbps at the same time so could have 160Gbps output all at once?

No port gives more than ≈22-25 Gbps of data (PCIe or USB) so the max total can't be more than ≈80Gbps.
The rest of the 40 Gbps of a port can only be used by DisplayPort data.

Each Thunderbolt port in Apple Silicon belongs to a separate Thunderbolt bus. This means for the M1 Max Studio, you can connect four displays spread among 2 to 4 Thunderbolt ports without regard to which ports are used.

For Intel Macs, there's usually a Thunderbolt bus for every two Thunderbolt ports. A pair of Thunderbolt ports belonging to the same Thunderbolt bus can connect two displays, either one per port or two from a single port.

 

[OSB]

For some additional clarity, the minimum spec for data over TB4 is 32Gb/s. But real world speeds will vary with controllers, cables, etc., and 25gb/s is where most drives seem to top out.

 

[joevt]

For some additional clarity, the minimum spec for data over TB4 is 32Gb/s. But real world speeds will vary with controllers, cables, etc., and 25gb/s is where most drives seem to top out.

I think 25 Gbps exceeds all drives (I haven't seen any benchmarks that get 3125 MB/s). Some drives may exceed 24 Gbps (3000 MB/s). Most drives are around the 22 - 24 Gbps range (2750 MB/s).

32 Gbps (actually 31.5 Gbps) is the minimum upstream PCIe connection for the host Thunderbolt 4 controller. It has little to do with the data that is transferred downstream over the Thunderbolt cable.

Tiger Lake Thunderbolt 4 buses have two ports which have a combined PCIe upstream that is greater than 32 Gbps like Ice Lake Thunderbolt 3 buses. That's because Ice Lake and Tiger Lake have integrated Thunderbolt controllers (inside the CPU) and don't use a real PCIe connection though they are accessed as PCIe devices. M1 Thunderbolt 4 controllers are similar but each bus has one port.

 

[OSB]

32 Gbps (actually 31.5 Gbps) is the minimum upstream PCIe connection for the host Thunderbolt 4 controller. It has little to do with the data that is transferred downstream over the Thunderbolt cable.

Interesting. In that case I've been misunderstanding that spec. I thought TB4 just replicated the same link over cable.

 

[joevt]

Interesting. In that case I've been misunderstanding that spec. I thought TB4 just replicated the same link over cable.

Thunderbolt PCIe (upstream from host controller or downstream from peripheral controller) is usually 4 lanes at 8 GT/s (with 128b/130b encoding). For Thunderbolt 3, some PCs may have Thunderbolt host controllers using 2 lanes or may use 5 GT/s link rate (with 8b/10b encoding).

The Thunderbolt signal on the cable is 2 lanes (or one lane if the cable is broken) at either 10.3125 or 20.625 Gbps (with 64b/66b encoding). PCIe or USB or DisplayPort or other data is encapsulated inside Thunderbolt packets (like how http or smb is for ethernet).