Transmission of PCIe data over a Thunderbolt 3 cable is capped at 22 Gbps (2750 MB/s) which is less than the max allowed by the Thunderbolt controller's PCIe 3.0 x4 connection.
PCIe expansion is more expensive (the cables) but isn't capped like Thunderbolt. If you use a PCIe 3.0 x4 upstream cable, then you get PCIe 3.0x4 speed. You can get cables up to PCIe 3.0x16 (126 Gbps). There is no bandwidth used for DisplayPort.
My solution with the amfeltec, if I could get it to actually work without errors, would be most similar to a PCIe expansion box with PCIe 3.0x16 upstream and four PCIe 3.0x4 slots. The difference is where the cable lengths are. In my case the upstream cable is 0 m (direct connect to the Mac Pro's PCIe slot), and the downstream port cables are 0.5 m (from the M.2 adapters). With a PCIe expansion box, the upstream cable is 1.5m and the slots are 0 m. The cables of the PCIe expansion box are easier to use - I had to do some horrible things with the M.2 adapter cables from the amfeltec - not to mention the hole I put in the amfeltec's back plate. I attached a picture of the parts:
View attachment 794463
I think I would have had better luck using a PCIe x16 riser cable, connect the amfeltec to that, then used non-cabled M.2 to PCIe x4 adapters. The PCIe 2.0 signals to the amfeltec can probably survive better over a cable than the PCIe 3.0 signals from the M.2 slots.
hi there!
after a lot of goofing around, I went the Pcie expension way !
first with gen2x8 on a netstor 16 bay rack, then on a gen2x16 cyclone microsytem 5 slot ATX expender because I hated loosing 8 lane on my second 16x slot of the macpro. This one was placed in a SC848 chassis with 24 bay and a redundent 1200w psu.
the cool part was that i could had my anfeltec in slot 1, the expender host card in slot 2, and in the expender my 3 P4000, my raid card, and a free slot, a usb3 card in slot 3, and a 10Gbe card in slot 4.
the only problem was the sc846 chassis is loud as ****... no kidding I mean really loud!!! And the heat generated by the 3 P4000 the raid card and 24 sas drive is just stupid.
So because I’m an idiot I bought two 3.1 xserve for 200€, have slaped them in a 12U rolling enclosure and my data will be hosted on one xserve with a 10gbe card, running the sc846 the raid.
For the second xserve, i have found a cheap nvidia tesla 4 GPU enclosure with the host card and 2 cable, it will have 4 P4000 gpu and will only be a dedicated video converter/exporter for premiere.
the 12U chassis will stay ine the basement.
Now I am gutting out an old Powermac G5 to have my new 10 slot cyclone expender in a silent and « MacPro look »
enclosure siting next to my 4.1.
this will have 4 P4000, and as much nvme drive on regular Kryo M2 adapter,
to build a fast scratch drive.
over the 10Gbe network I get consistent 700/900 Mb/s read write to the xserve.
and as my preview will be on the nvme array, i think i will be ok.
the good thing is those cyclone micro systems expender are cheap to buy if you know witch one to look for.
but you have to know what you are buying because some board are only 4x some are 8x and a very few are 16x.
some are Gen1....
any how, if you have the right host/cable/backplane, they are basically a straight plug an play.
I am still experimenting with the tesla enclosure, because it needs every slot populated to turn on, but other than that it is just a 4x16 epender in a 1u chassis with it’s onwn psu. the obsolete tesla card can be replaced by newer one, but you have to find card with a low TDP such as a P4000 because the psu is only 900w max so 4x200w max peak.