Just put the GPUs on PCIe 3.0 x8 slots, and you have plenty of extra slots to use for TB.
You know that's what Apple has done....
No, I don't know. Frankly, if there wasn't a PCIe SSD it wouldn't be required. Anyway, there is absolutely no reason to drop
both GPUs down to x8.
...
Also, there are 8 lanes of PCIe 2.0 - which would be fine for T-Bolt.
Pragmatically, there are not. The Mac Pro has two Ethernet sockets. USB 3.0 ports , and Wifi/Bluetooh. All of those eat into the PCIe v2.0 lane budget.
What is included in the system is
2 Ethernet ports ===> two 1x PCIe v2.0
USB 3.0 ===> one 1x PCIe v2.0
Wifi/Bluetooth ===> one 1x PCIe v2.0
That's is four PCIe v2 lanes right there. ( presuming Audio is fully leveraged off the chipset and won't need one. Otherwise pick one of the above to split on a switch. ).
Add a TB controller (x4) and the chipset's lanes are all gone. [ can play games by adding PCIe SSD here but still all gone if it is a x4 v2 device ]
Still have two TB controllers. E5 Xeon has 40. So.
GPU daughtercard ==> x16 PCIe v3.0
GPU daughtercard ==> x16 PCIe v3.0
Thunderbolt Controller ==> two x4 PCIe v2.0
That's 40. It would actually all fit. But the the SSD runs that over.
If it is a x4 PCIe v2 SSD then one of the four x4 PCIe v2.0 has to pair up or need to "borrow" from one of the x16 bundles.
If there is a switch which can trunk down x4 PCIe v2.0 into x2 PCIe v3.0 worth of utilization then don't really need to gut the GPUs. Likewise a switch that doesn't drop all sides down to v2.0 if just one is. So for example a:
x16 PCI v3.0 by ( x16 PCI v3.0 by x4 PCIe v2.0 ) switch. (probably somewhere near x10-12 throughput depending upon how well juggle around the x4 v2.0 )
Or
x4 PCIe v3.0 by ( x4 PCI v3.0 + x4 PCIe. v2.0 ) switch.
Both would have less than x8 impact on the GPU cards. Although in the second case the SSD and one of the thunderbolt networks would have major impact in concurrent use.
Worst case the E5 is configured x16 , x8 , x8 , x4 , x4 And just use a
x8 PCIe by ( x4 PCI + x4 PCI ) switch
to crank out four x4 bundles ( leaving door open to perhaps have dual SSDs in future configurations ). Could move all three TB controllers and SSD here. The vacated chipset lanes could go to connection to power management (typically a PCI bridge ) and other misc internal management duties. ( also audio if necessary). If Apple wanted to maximize USB 3.0 throughput they could use two 2 port/socket USB 3.0 controllers to provision the 4 sockets.
This is far more a technology mismatch. In terms of bandwidth the SSD and TB controllers really only need x2 worth of PCIe v3. If the SSD is a v2 device it is drag on maximum E5 PCIe controller utilization as much as the TB controllers are. A x2 PCIe v3 SSD would put less "borrowing" pressure on the set-up.
