Thunderbolt 3 M.2 NVMe SSD Enclosures

[qwrty]

Hello :]

Anyone running Crucial P5 Plus 1Tb or 2Tb version with a TBU401 or TBU405 ?
Cannot find any benchmark with this enclosure.

P5 Plus seems to be a real alternative versus 980 Pro, but don't know about performance inside the enclosure...

Thx

 

[whodiini]

Is there anyone who knows how to update firmware for a JHL6340 based enclosure to the latest v47?

Mine is on 41.1 and I have intermittent dropouts. My manufacturer Orinco does not provide any support nor do they respond to any email in any form at any place

 

[Water99]

After I update my Mac Studio to Monterey 12.5.1, My Orico USB4.0 Enclosure M2V01 and SN750 1TB seem to boost up a lot, I really mean it, A LOT.

AmorphousDiskMark test:
Before update: Read: 2300 mb/s Write: 2300 mb/s (Black magic read about 2700 mb/s write 2200 mb/s)
Updated: Read: 3100mb/s write: ~2600 mb/s (black magic read 2700 mb/s write ~2500 mb/s)

https://imgur.com/RMLyvJg

I have watched all the review videos, and I know that Orico M2V01 and Acasis TBU401 are the same about performance but Orico is hotter while using, and about SN750, I also know the 1TB is much better than 2TB about write speed.

The video about performance ORICO VS ACASIS:

 

[MediaGary]

In my experience, I've always get slightly boosted/cached/impossible 'Read' speeds from the Amorphous Disk Benchmark, when compared to the Blackmagic Disk Speed Test. For completeness, the ATTO Disk Benchmark test also shows some of the boosted/cached performance effects in the larger transfer sizes.

 

[joevt]

In my experience, I've always get slightly boosted/cached/impossible 'Read' speeds from the Amorphous Disk Benchmark, when compared to the Blackmagic Disk Speed Test. For completeness, the ATTO Disk Benchmark test also shows some of the boosted/cached performance effects in the larger transfer sizes.

Not sure what you mean by impossible. All the numbers I've seen are within the theoretical limits of the connection. 3100 MB/s for Thunderbolt, 1060 MB/s for USB 3.1 gen 2.

PCIe gen 3 limit is 3938 MB/s. USB limit is 10 Gbps = 1212 MB/s. Those are without protocol overhead.
An NVMe in a PCIe slot might be able to get up to 3500 MB/s. The Thunderbolt benchmark doesn't get that high so it seems reasonable.

 

[MediaGary]

Not sure what you mean by impossible. All the numbers I've seen are within the theoretical limits of he connection. 3100 MB/s for Thunderbolt, 1060 MB/s for USB 3.1 gen 2.

PCIe gen 3 limit is 3938 MB/s. USB limit is 10 Gbps = 1212 MB/s. Those are without protocol overhead.
An NVMe in a PCIe slot might be able to get up to 3500 MB/s. The Thunderbolt benchmark doesn't get that high so it seems reasonable.

Caching effects occasionally happen: I've seen peak values of ~810MByte/s in ATTO and Amorphous benchmark figures across my 10GbE network that connects to one of my Win10 Pro "servers". That particular server has a 4-member RAID-0 made of 4TB HGST HDDs. Within the server, not traversing the network, the typical figure accomplished is the expected ~600MByte/s.

I'll try to get the time to run all three benchmarks against the same set of four drives for my 2010 Mac Pro, and will post the results. The four drives are: 2TB Sabrent NVMe on Sonnet 4x4, 8TB 2-member RAID-0 NVMe on Sonnet 4x4, 17TB fusion drive, and the network connected RAID-0 mentioned above. BTW, there's a second network-connected 42TB archive "server" is not in the test plan.

 

[whodiini]

Here's a discovery on thermal control: External NVMe enclosure gets very hot and at some point, thermal throttling occurs. Found a simple solution - place the external enclosure (flat face down) on top of your mac mini or mac studio, The aluminum case of the enclsoure conducts the heat to the big block of aluminum of the computer. It lowers the temperature by more than 20 degree C! Dont even need a thermal pad. Simple, zero cost, very effective solution!

 

[Rb70]

I bought a Trebleet Thunderbolt 3 enclosure that takes two NVMe drives. I was thinking of putting two 4tb sticks in it but am unsure what to buy. The Sabrent 4tb look really good but Trebleet strongly discourages their use on Macs. Apparently there are issues with some of the NVMe. I intend to use the enclosure with my Mac Studio Ultra for video and audio editing.
Is anyone using Sabrent NVMe with a Mac Studio? Any recommendations or thoughts?

 

[theorist9]

I bought a Trebleet Thunderbolt 3 enclosure that takes two NVMe drives. I was thinking of putting two 4tb sticks in it but am unsure what to buy. The Sabrent 4tb look really good but Trebleet strongly discourages their use on Macs. Apparently there are issues with some of the NVMe. I intend to use the enclosure with my Mac Studio Ultra for video and audio editing.
Is anyone using Sabrent NVMe with a Mac Studio? Any recommendations or thoughts?

What about the Crucial P3 Plus? At ~4000–5000 MB/s, it might saturate the interface just as well as a faster SSD, but at $330/4 TB, it's a much better value. Trebleet hasn't tested them, so you'd want to buy them somewhere you have return privileges. Unfortunately, they're out of stock everywhere now (at least the 4 TB size), but Crucial is working on bringing in more.

At the other end of the spectrum is the WD SN850X (~$550/4TB) which (based on the performance of the SN850), should be one of the fastest drives around. And it's current available. But, again, Trebleet has not yet tested it. And hopefully it doesn't require a firmware update like the SN850 does (or did).

 

[mackiemesser2]

@joevt

"40 Gbps is really 32.4 Gbps for non-video data

Of the ~40 Gbps = 5 GB/sec bandwidth of Thunderbolt 3, ~8 Gbps can be used ONLY for video data, leaving a maximum theoretical bandwidth for non-video data of 32.4 Gbps = 4.05 GB/sec.

Thus the maximum theoretical bandwidth for non-video data is 32.4 Gbps = 4.05 GB/sec. However, that data is 8b/10b encoded, so the true *usable* bandwidth drops to 25.92 Gbps ~= 3.24 GB/sec. Good luck with that—MPG has never been able to test any Thunderbolt 3 SSD at faster than about 2.645 GiB/sec= 2.84GB/sec. So much for specifications versus achievable real-world data throughput."

Understanding Thunderbolt 3 Bandwidth

I think all the benchmarks >3GB/s are lying

 

[joevt]

@joevt
"40 Gbps is really 32.4 Gbps for non-video data

Of the ~40 Gbps = 5 GB/sec bandwidth of Thunderbolt 3, ~8 Gbps can be used ONLY for video data, leaving a maximum theoretical bandwidth for non-video data of 32.4 Gbps = 4.05 GB/sec.

Thus the maximum theoretical bandwidth for non-video data is 32.4 Gbps = 4.05 GB/sec. However, that data is 8b/10b encoded, so the true *usable* bandwidth drops to 25.92 Gbps ~= 3.24 GB/sec. Good luck with that—MPG has never been able to test any Thunderbolt 3 SSD at faster than about 2.645 GiB/sec= 2.84GB/sec. So much for specifications versus achievable real-world data throughput."

Understanding Thunderbolt 3 Bandwidth

I think all the benchmarks >3GB/s are lying

That macperformanceguide.com article was full of inaccuracies and I told him that. He did not attempt the tests I suggested or make corrections. He misinterpreted some crappy Intel marketing material for some of his statements rather than doing tests or doing real research.

Thunderbolt transmits on the wire at 41.25 Gbps. It uses 64b/66b encoding so the actual amount of data is 40 Gbps. Thunderbolt is the only modern connection I know where the data bandwidth (40 Gbps) rather than the wire bandwidth (41.25 Gbps) is used to describe the bandwidth for normal users. USB 3.0 is actually 4 Gbps of data, not 5 Gbps. USB 3.1 gen 2 is actually 9.7 Gbps of data, not 10 Gbps. Same for SATA (6G -> 4.7 Gbps), HDMI (18 Gbps -> 14.4 Gbps), etc.

DisplayPort is tunnelled over Thunderbolt so the 8b/10b encoding used by DisplayPort is not a factor since Thunderbolt has its own 64b/66b encoding.

Bandwidth used by a display is pixel clock (MHz) x bits per pixel (bpp).
2.5K 2560x1600 269MHz * 30bpp = 8 Gbps.
4K 3840x2160 533MHz * 30bpp = 16 Gbps.
5K 5120x2880 483.37MHz * 2 * 30bpp = 29 Gbps.
6K 6016x3384 648.91MHz * 2 * 30bpp = 38.9 Gbps.

The 5K and 6K cases use a dual tiled mode which uses two DisplayPort connections, one for each half of the display. In the case of 6K, the connections are both HBR3 which would total 51.84 Gbps but it works because not all of the DisplayPort HBR3 bandwidth is required for the two 3008x3384 halves and Thunderbolt does not transmit the DisplayPort stuffing symbols used by DisplayPort to fill the DisplayPort link bandwidth.

The 40 Gbps is not shared between ports. Each port can do 40 Gbps for 80 Gbps total but it's currently impossible to transmit 80 Gbps of data+video. The Thunderbolt controller of a host has an upstream PCIe connection to the host bus and two upstream DisplayPort connections to the host GPU. The Thunderbolt controller has two downstream Thunderbolt ports. 38.9 Gbps of video can be output to one port while the other port transmits 22-25 Gbps of data. ≈60 Gbps total.
Using two HBR3 displays, you can transmit ≈25 Gbps two each port, and transmit ≈11 Gbps to each port. ≈72 Gbps total.

The upstream PCIe connection is usually PCIe 3.0 x4 = 31.5 Gbps. The max amount of data I was able to get using two ports of a discrete Thunderbolt controller (Alpine Ridge or Titan Ridge) was ≈23 Gbps. Integrated Thunderbolt controllers (Ice Lake, Tiger Lake, M Series Macs) don't have the PCIe 3.0 x4 upstream limit.

Newer GPUs support DSC which can be used by new displays such as the Apple Studio Display or Apple Pro Display XDR to reduce average bpp to 12.
5K 960MHz * 12bpp = 11.52 Gbps.
6K 1286MHz * 12bpp = 15.4 Gbps.
So a 6K display can use less bandwidth than a 4K display.

DisplayPort is mostly transmit. Thunderbolt has separate lines for transmit and receive, therefore, while a display may eat into data transmit bandwidth, the data receive bandwidth is wide open.

I don't think the 24 Gbps benchmarks are lying. I think slight adjustments have been made to Thunderbolt peripheral firmware since the first Alpine Ridge devices. Also, if you think that disk benchmarks are being affected by caching, then consider non-disk related benchmarks, such as for eGPUs. I think I've seen some of those approach or exceed 3000 MB/s.
https://egpu.io/forums/builds/2019-...3-mastercooler-eg200-macos-12-1-win11-itsage/

 

[mackiemesser2]

That was very informative and I believe you. Do you have any source that is not that "crappy Intel marketing material"?
All I got since Thunderbolt3 came out was that information and honestly all my knowledge was based on this, wikipedia (sigh) or other readings here and there. Thanks Intel for these pdf's....

Just to recap for myself:

Thunderbolt 3/4 = 41.25 Gbps > 64b/66b encoding > 40 Gbps per port = 80 Gbps total

80Gbps (theoretical)

Upstream:
TB3: PCIe 3.0 x4 = 31.5 Gbps + 2x DP1.2 (32,4 Gbps HBR3)
TB4: PCIe Bus > PCIe 3.0 x4 ≈40 Gbps (max?) + 2x DP1.4 (32,4 Gbps HBR3 + DSC 1.2)


Downstream:
2x TB3/ TB4 ports

either

38.9 Gbps video (should be 32,4 Gbps?) + *22-25 Gbps data* ≈60 Gbps total

or

2x HBR3 = ≈25 Gbps video per port + *≈11 Gbps per port* ≈72 Gbps total


If you only could replace the ≈ with = here and there


I always was under the impression that either ≈60 Gbps total or ≈72 Gbps total was the max limit. But the detail seems to lie in *22-25Gbps (2875Mbs - 3125Mbs)* that's what differs between TB3 and TB4 then. (That should also differ for 2x HBR3, right?)

The upstream PCIe connection is usually PCIe 3.0 x4 = 31.5 Gbps. The max amount of data I was able to get using two ports of a discrete Thunderbolt controller (Alpine Ridge or Titan Ridge) was ≈23 Gbps

My only "defense" is that I'm still on an Alpine Ridge which has that PCIe 3.0 x4 = 31.5 Gbps limit. So in my case the ≈23 Gbps alway was the "real world" limit. However I did neglect that integrated TB4 controllers now don't have that limit.

Sorry about that.

 

[joevt]

That was very informative and I believe you. Do you have any source that is not that "crappy Intel marketing material"?
All I got since Thunderbolt3 came out was that information and honestly all my knowledge was based on this, wikipedia (sigh) or other readings here and there. Thanks Intel for these pdf's....

The USB4 spec might be interesting. Mostly everything else I got from doing tests and benchmarks or reading about tests and benchmarks made by other people.

Just to recap for myself:

Thunderbolt 3/4 = 41.25 Gbps > 64b/66b encoding > 40 Gbps per port = 80 Gbps total

Downstream:
2x TB3/ TB4 ports

80Gbps (theoretical)

Thunderbolt controllers for Alpine Ridge (TB3), Titan Ridge (TB3), Maple Ridge (TB4), Ice Lake (TB3), Tiger Lake (TB4) have two ports per Thunderbolt controller (called a Thunderbolt bus on Macs).

M Series Macs have one Thunderbolt port per controller but still support two DisplayPort connections per controller.

Upstream:
TB3: PCIe 3.0 x4 = 31.5 Gbps + 2x DP1.2 (32,4 Gbps HBR3)
TB4: PCIe Bus > PCIe 3.0 x4 ≈40 Gbps (max?) + 2x DP1.4 (32,4 Gbps HBR3 + DSC 1.2)

- 2x HBR2 for Alpine Ridge.
- Only the newer Thunderbolt controllers (Titan Ridge and later) support HBR3.
- My Mac mini with Titan Ridge is limited to HBR2 because of the GPU.
- I think Maple Ridge (TB4) has similar PCIe limits as TB3.
- The 32.4 Gbps number for HBR3 is 8b/10b encoding which is not transported on the Thunderbolt cable. The data it encodes is 25.92 Gbps. The input DisplayPort signal to the host Thunderbolt controller is 32.4 Gbps and the output signal from the Thunderbolt controller of the peripheral device will also be 32.4 Gbps.
- DSC is a property of the DisplayPort signal. I don't think the Thunderbolt controller has any affect on that (so Alpine Ridge can output HBR2 DSC).
- The upstream of integrated Thunderbolt controllers (Ice Lake, Tiger Lake, M Series Macs) is not real PCIe and is therefore not limited to PCIe 3.0 x4. Raiding two NVMe devices, one per Thunderbolt port, can get ≈23Gbps total for discrete Thunderbolt controllers and ≈38Gbps for integrated Thunderbolt controllers like Ice Lake. I don't recall if similar tests were done on Tiger Lake or M Series. I believe the 38Gbps for integrated Thunderbolt has no relationship to the 40 Gbps connection of Thunderbolt.

either

38.9 Gbps video (should be 32,4 Gbps?) + *22-25 Gbps data* ≈60 Gbps total

38.9 Gbps is for the Apple Pro Display XDR 6K60 10bpc when used with a GPU that supports HBR3 but not DSC.
32.4 Gbps is for the input or output DisplayPort HBR3 signal outside the Thunderbolt controllers. The tunnelled DisplayPort data that exists on the Thunderbolt cable between two Thunderbolt controllers is is up to 25.92 Gbps (depends on the pixel clock and average bits per pixel). Encapsulating DisplayPort data as tunnelled Thunderbolt packets must had some bytes. Same for tunnelled PCIe...

or

2x HBR3 = ≈25 Gbps video per port + *≈11 Gbps per port* ≈72 Gbps total

If you only could replace the ≈ with = here and there

There's overhead to account for, the speed of your NVMe, the benchmark used, etc. I suppose for DisplayPort you can get pretty close to 25.92 Gbps. You can increase the pixel clock in a custom resolution timing by .1 MHz at a time until you hit the limit. I don't know of a method to do that with M series Macs.

I always was under the impression that either ≈60 Gbps total or ≈72 Gbps total was the max limit. But the detail seems to lie in *22-25Gbps (2875Mbs - 3125Mbs)* that's what differs between TB3 and TB4 then. (That should also differ for 2x HBR3, right?)

I'm not to clear on the difference between TB3 and TB4. If one can get 3000 MB/s then the other should be able to as well. But yeah, DisplayPort maxing is easy. Maxing PCIe data is fuzzy. Also with TB4, the Thunderbolt signal can include tunnelled USB, not just tunnelled PCIe and tunnelled DisplayPort. So there might be interesting things with adding tunnelled USB into the mix.

My only "defense" is that I'm still on an Alpine Ridge which has that PCIe 3.0 x4 = 31.5 Gbps limit. So in my case the ≈23 Gbps alway was the "real world" limit. However I did neglect that integrated TB4 controllers now don't have that limit.

The max bandwidth of integrated Thunderbolt controllers is not much of an advantage over discrete Thunderbolt controllers. 31.5 Gbps is far enough away from the 23 Gbps that it's probably not the bottleneck. Even the 25 Gbps that some people have seen is not that close to the 31.5 Gbps limit. Consider that an NVMe in a real PCIe 3.0 x4 slot is supposed to get 28 Gbps. 23 Gbps is about the max I can get with my Titan Ridge and the NVMe drives I have.

Maybe the big advantage with integrated Thunderbolt controllers is latency since they're built into the CPU and don't need to go through a real PCIe connection.

 

[theorist9]

I don't think the 24 Gbps benchmarks are lying. I think slight adjustments have been made to Thunderbolt peripheral firmware since the first Alpine Ridge devices. Also, if you think that disk benchmarks are being affected by caching, then consider non-disk related benchmarks, such as for eGPUs. I think I've seen some of those approach or exceed 3000 MB/s.
https://egpu.io/forums/builds/2019-...3-mastercooler-eg200-macos-12-1-win11-itsage/

According to OWC, 2800 MB/s is the upper data transmission limit for any TB interface (there's also separate bandwidth reserved for video; but here they are talking about data only, where "data" means non-video signals, i.e., what you'd transmit between a host and an external drive).

OWC 4.0TB Envoy Pro FX Thunderbolt (40Gb/s) + USB... at MacSales.com

TB3ENVPFX04 4.0TB OWC Envoy Pro FX Thunderbolt (40Gb/s) + USB 3.2 (10Gb/s) Bus-Powered Portable NVMe SSD External Storage Solution. The fastest, most compatible portable drive ever made that works with yesterday's, today's and tomorrow's Thunderbolt and USB computers and devices.

eshop.macsales.com

 

[joevt]

According to OWC, 2800 MB/s is the upper data transmission limit for any TB interface (there's also separate bandwidth reserved for video; but here they are talking about data only, where "data" means non-video signals, i.e., what you'd transmit between a host and an external drive).

Right. The original number from Intel was 22 Gbps = 2750 MB/s. Then I saw these 2800 MB/s numbers which is 22.4 Gbps which I thought was just 2750 MB/s rounded up and I asked how they got those numbers and they didn't say. Then 6 years later we're seeing 2900 MB/s, 3000 MB/s = 24 Gbps, and sometimes 25 Gbps = 3125 MB/s. I don't think anyone's gotten over 26 Gbps = 3250 MB/s yet.

Bandwidth is not reserved for video. Rather, video takes whatever bandwidth it needs, and whatever remains can be used for PCIe and USB up to a certain limit (which is < 26 Gbps so far as we know). Apple Pro Display XDR takes up to 38.9 Gbps leaving only 1 Gbps for its USB 5 Gbps ports but only for transmit - receive should still allow 5 Gbps (Apple says USB ports are reduced to USB 2.0 in this case, but I haven't seen anyone test for either case).

 

[theorist9]

Right. The original number from Intel was 22 Gbps = 2750 MB/s. Then I saw these 2800 MB/s numbers which is 22.4 Gbps which I thought was just 2750 MB/s rounded up and I asked how they got those numbers and they didn't say. Then 6 years later we're seeing 2900 MB/s, 3000 MB/s = 24 Gbps, and sometimes 25 Gbps = 3125 MB/s. I don't think anyone's gotten over 26 Gbps = 3250 MB/s yet.

Bandwidth is not reserved for video. Rather, video takes whatever bandwidth it needs, and whatever remains can be used for PCIe and USB up to a certain limit (which is < 26 Gbps so far as we know). Apple Pro Display XDR takes up to 38.9 Gbps leaving only 1 Gbps for its USB 5 Gbps ports but only for transmit - receive should still allow 5 Gbps (Apple says USB ports are reduced to USB 2.0 in this case, but I haven't seen anyone test for either case).

Yes, it is confusing. The bus-powered NVMe enclosures from the more established companies (Sabrent, OWC) can all do 1553 MB/s over 2 PCIe channels. If you extrapolate to four, that would give 1553*2 MB/s.008 Gb/MB =24.848 Gb/s.* But I don't know if it really doubles for four.

*There's an interesting reason why they limit their bus-powered enclosures to 2 lanes, which I offer in my next post.

Bandwidth is not reserved for video. Rather, video takes whatever bandwidth it needs, and whatever remains can be used for PCIe and USB up to a certain limit (which is < 26 Gbps so far as we know). Apple Pro Display XDR takes up to 38.9 Gbps leaving only 1 Gbps for its USB 5 Gbps ports but only for transmit - receive should still allow 5 Gbps (Apple says USB ports are reduced to USB 2.0 in this case, but I haven't seen anyone test for either case).

It's my understanding that 8 Gbps is reserved for video, leaving a theoretical max of 32 Gbps for data.

 

[theorist9]

Is it safe to get these super-fast TB3 NVMe enclosures—ones capable of ~2800 MB/s? As far I can see, unlike the cases from established vendors like OWC and Sabrent (which are limited to ~1550 MB/s), these aren't TB-certified.

Details:

As I understand it, NVMe SSD’s can exceed the power draw supplied by the TB bus, resulting in a disconnection during data transfer, which can lead to data loss or corruption. Consequently, TB has power limits for bus-powered enclosures, and to get TB3 certification you need to show your system won't exceed those. Here's a screenshot from OWC's Envoy Express page:

If you are selling a TB3 enclosure/drive combo (like the Samsung X5), you just need to design your SSD not to exceed those limits, and can use all four PCIe lanes, giving high speeds.

But if you want to sell a bare enclosure, you don't know which SSD will be installed. In that case, supposedly the only way for a vendor to guarantee they won't exceed the power limits for a bus-powered case, and thus get TB certification, is to limit the enclosure to two lanes (halving the power draw). That's why established vendors, whose devices are TB-certified, use only two lanes, which limits the max speed of their enclosures to 1553 MB/s. Again for the OWC:

And yet there's one piece of this that doesn't add up: If you take, say, the Sabrent dual NVMe enclosure, which is both bus-powered and TB-certified, and run the drives in RAID 0, you will get up to 2500 MB/s (according to Sabrent). That would need to use all four lanes, and I don't understand why two SSDs in RAID 0 over 4 PCIe lanes would draw less peak power than a single SSD over 4 PCIe lanes.

Anyone care to enlighten me?

 

[joevt]

It's my understanding that 8 Gbps is reserved for video, leaving a theoretical max of 32 Gbps for data.

I don't think there's a point to making a reservation for something that might not get used. Well, no-one can get over 26 Gbps of data so the idea isn't useful. Also, 32 Gbps isn't even attainable from a single Thunderbolt device since Thunderbolt 3/4 controllers only do PCIe gen 3 x4.

Here's a test:

1. Start with a Thunderbolt host controller that doesn't have a PCIe 3.0 x4 upstream limit (31.5 Gbps). Those controllers includes only integrated Thunderbolt controllers (Ice Lake, Tiger Lake, M Series Macs). I don't know exactly what kind of upstream each connection has. I know Ice Lake can do ≈38 Gbps with PCIe data from both ports of the controller. M series controllers only have one port per bus.
2. Then connect a Thunderbolt device that has two Thunderbolt ports with NVMe to get 2000 MB/s.
3. Then connect a second Thunderbolt device with NVMe that can also get 2000 MB/s.

Can this combination get more than 26 Gbps (3250 MB/s) using RAID 0 or ATTO Disk Benchmark.app? It's technically possible since the Thunderbolt cable can do 5000 MB/s. It depends if the tunnelled PCIe from the Thunderbolt devices in the chain needs to be converted down to real PCIe gen 3 link rate * link width before it gets to the host controller.

 

[joevt]

And yet there's one piece of this that doesn't add up: If you take, say, the Sabrent dual NVMe enclosure, which is both bus-powered and TB-certified, and run the drives in RAID 0, you will get up to 2500 MB/s (according to Sabrent). That would need to use all four lanes, and I don't understand why two SSDs in RAID 0 over 4 PCIe lanes would draw less peak power than a single SSD over 4 PCIe lanes.

Anyone care to enlighten me?

The Thunderbolt controller in a Thunderbolt device acts as a PCIe bridge and can be setup to support these combinations of PCIe lanes:
x4
x2x2
x2x1x1
x1x1x1x1

A normal NVMe enclosure uses x4 to support 1 NVMe device for ≈2800 MB/s.
The Sabrent dual NVMe enclosure uses x2x2 mode to support 2 NVMe devices. Each M.2 slot can do ≈1550 MB/s.
The OWC Express 4M2 uses x1x1x1x1 mode to support 4 NVMe modules and is not bus powered. Each M.2 slot can do ≈775 MB/s.
A Thunderbolt 3 dock can use x1x1x1x1 for two USB 3.0 controllers, a Ethernet controller, and a Fire Wire controller, or a ASM1142 USB controller.
A Thunderbolt 3 dock with Titan Ridge or a Thunderbolt 4 dock (Goshen Ridge) usually won't use their PCIe lanes so that their devices (all USB) can be used when the dock is connected to a non-Thunderbolt host.
Actually, CalDigit TS4 is the only Thunderbolt 4 dock I know that uses its PCIe lane - for the Ethernet controller - which makes the Ethernet controller not work for non-Thunderbolt hosts. Goshen Ridge only has one PCIe lane for PCIe devices.

Now, if you want multiple M.2 devices to get x4, then you need a Thunderbolt enclosure with a PCIe bridge that has 12 or more lanes to divide between the upstream connection (Thunderbolt's gen 3 x4) and the downstream M.2 connections (gen 3 or gen 4, 4 lanes each). The Sonnet Echo Express III-D has 3 slots that each can do at least x4. Or you can get a Thunderbolt enclosure with a single PCIe slot and install a PCIe card that has a PCIe bridge and multiple M.2 slots (up to 8 for some cards). Or you can combine both to tripple the number of M.2 slots that a single PCIe card can support - so up to 24 NVMe total, though you could add PCI bridges on top of PCI bridges.... but in that case you might as well setup a NAS with 100 Gbps connection...

As for how a dual NVMe enclosure can be bus powered, they can't. The product page for the Sabrent dual NVMe enclosure says "Power adapter is required and included."

 

[theorist9]

I don't think there's a point to making a reservation for something that might not get used.

Forgive me if I'm misunderstanding you, but it sounds like your statement is about what the standard should be. But I thought we were discussing was the standard actually is. And for that, it's my understanding that it does indeed reserve 8 Gbps for video data only (it can of course use more if needed). If so, 1/4 of its total bandwidth is unavailable for data because of this.

It seems it shouldn't be that hard to confirm this either way....

 

[joevt]

Forgive me if I'm misunderstanding you, but it sounds like your statement is about what the standard should be. But I thought we were discussing was the standard actually is. And for that, it's my understanding that it does indeed reserve 8 Gbps for video data only (it can of course use more if needed). If so, 1/4 of its total bandwidth is unavailable for data because of this.

It seems it shouldn't be that hard to confirm this either way....

Where does it say 8 Gbps is reserved for video? I think it's more likely that 8 Gbps cannot be used for data (because the discrete Thunderbolt 3 controllers were limited to 31.5 Gbps) which has a different meaning.

It's not much of a standard if no-one is allowed to read it. For that you might try the USB4 spec which is open.

 

[theorist9]

As for how a dual NVMe enclosure can be bus powered, they can't. The product page for the Sabrent dual NVMe enclosure says "Power adapter is required and included."

Ah! Mystery solved!

A normal NVMe enclosure uses x4 to support 1 NVMe device for ≈2800 MB/s.

Those are the ones you see from ACASIS, Trebleet, ORICO, Yottamaster, etc., and they're all bus-powered. And I don't think any of them are TB-certified. Do they risk disconnection (by exceeding the power limit on the TB port) if you put in a higher-performance (and thus higher-powered) NVMe drive?

Why doesn't anyone make a 2800 MB/s wall-powered version of these single-NVMe enclosures that, as a result of not relying on bus power, avoids this problem? Sabrent and OWC only make non-powered 2-lane versions limited to 1500 MB/s, and while Glyph makes a powered enclosure, it's also limited to 1500 MB/s:

Glyph Thunderbolt 3 NVMe Dock (0 GB)

Glyph Thunderbolt 3 NVMe Dock (0 GB)

www.amazon.com

 

[theorist9]

Where does it say 8 Gbps is reserved for video? I think it's more likely that 8 Gbps cannot be used for data (because the discrete Thunderbolt 3 controllers were limited to 31.5 Gbps) which has a different meaning.

It's not much of a standard if no-one is allowed to read it. For that you might try the USB4 spec which is open.

I don't recall the source for the specific 8 Gbps. But a tech lead from a US Thunderbolt hub company told me that some bandwidth is preallocated for video in the TB standard.

 

[joevt]

some bandwidth is preallocated for video in the TB standard.

Why is it preallocated? How much is preallocated? What does that affect? What do you have to do to make that affect something? Is this preallocation carried over into the USB4 spec? If so then maybe your tech lead can point to that.

 

[theorist9]

Why is it preallocated?....What does that affect? What do you have to do to make that affect something? Is this preallocation carried over into the USB4 spec? If so then maybe your tech lead can point to that.

No idea. And given all the confusion, it seems the only way to get solid info. would be to ask Intel. I'm actually thinking it might be worth emailing them about these and the other questions and seeing if I get an answer.

How much is preallocated?

Again, the figure I recall seeing is 25% of the bandwidth. But I don't have a source.

Are you able to have any better luck answering my questions?