Why Thunderbolt
This essay starts on the premise that mobile computing is the trend, even amongst creative professionals, and that the PC is in decline, and along with it, vendor interest in developing and supporting PCIe expansion cards for PCs. And we all know that the Mac Pro workstation market is a tiny fraction of that. Those of us who have tried to update our Mac Pros also know how small that market is first hand thanks to the limited selection, poor vendor support, spotty drivers, and often silly compromises we've all come to expect as normal... It's ridiculous to me that we get one or two new GPUs every few years. It's insane that I need to run a power cable from my optical bay to power a USB 3 card because the only choice for native driver support is so poorly designed. It's maddening that finally 3 years after SATA3 SSDs became common, we could find a bootable SATA3 card that wouldn't bottleneck our drives. It's unimaginable to me that I might actually wish this state of affairs continue, never mind be passionately fighting (fruitlessly) for its continuation.
That brings me to Thunderbolt... What is it really? From a Mac Pro or PC perspective, it's just a PCIe x4 slot turned into a port. From a mobile computing perspective, that's huge... A pair of (switched) x4 PCIe slots on every Mac exposed as a small pair of ports. That takes mobile computing to a whole new level of expansion. And that means, there's a massive potential market for peripherals that plug into this port. A way bigger market than ever existed for PCIe cards for the Mac Pro. It's still early days, but this is a better place to be than in a declining market that couldn't garner much vendor interest.
Even PC motherboards and laptops are getting in on the game, and PC laptops have just as much to gain from TB as MacBooks and iMacs. This expands the market further and will attract even more vendors and create economies of scale to reduce prices. It's going to take some time but this will reach a critical mass soon enough.
The new Mac Pro
Now, how does Apple implement TB ports vs PCIe slots on the new Mac Pro?
There are three ways to implement TB (which carries DisplayPort video and x4 PCIe for data peripherals) and the three implementations are differentiated by how the DisplayPort signal gets to the TB controller...
1. IGP on a consumer/mobile CPU providing DisplayPort to the TB controller (Asus is going this route)
2. Cable kludge that routes DisplayPort from a discrete GPU to the TB controller (ASRock chose this)
3. Custom GPUs providing DisplayPort to the TB controller (Apple chose this)
Note that none of these precludes PCIe slots. However, in any of these implementations PCIe lanes are either dedicated to TB or switched/shared with other on-board PCIe peripherals/slots.
Apple chose to dedicate the PCIe lanes to internal peripherals (dual GPUs and at least one, maybe two PCIe SSDs, and integrated USB3 - the items most of us have added to our own Mac Pros) AND, on top of that, there are three TB controllers (6 ports) that have a x4 bus connection each.
While many don't/won't see this as a net gain, I do.
My current Mac Pro cannot have dual high-end GPUs, one or two PCIe SSDs, USB3, and three x4 PCIe peripherals all working at the same time. It simply can't do it. The new Mac Pro can. In fact, the new Mac Pro ships with more capability out of the box than you can possibly configure in the current Mac Pro (since a second high-end GPU covers one of the x4 slots) AND it allows you to add at least three x4 PCIe peripherals on top of that all while driving a trio of 4K displays.
Apple has chosen a direction which removes the Mac Pros expansion capabilities from the hands of the declining PC market (and all but dead Mac Pro PCIe card market) and puts it clearly in the hands of a market with orders of magnitude more promise... Every Mac sold in the last two generations has come equipped with TB ports and that is starting to get attention. For vendors, they can make a TB product and have a potential market of Millions of Mac owners. And now, Mac Pro owners are going to participate, and likely drive this market further (way more than a new Mac Pro with PCIe slots would) with their deep pockets and demanding requirements for performance .
Now despite the benefits of having more capability out-of-the-box and plenty of TB expansion, there's still the objection that you can't upgrade your GPU. Of course this is true, but here are a couple of considerations:
1. Upgrading the GPU in the past has not been widely adopted either... Limited choices offering much if any legit support, power supply constraints, expensive (over priced cards), etc... as a result, only a brave few would bother and even then, it only really makes sense to undertake this every two to three years anyway.
2. Upgrading the GPU in the past has not been warranted in many workflows and many are still productive after a few years with the GPU that shipped with their system.
3. Computers are systems and rarely does unlocking one bottleneck not reveal another immediately whereby the ultimate solution is an upgrade of several key components to really unlock new computing potential. At this point it's usually warranted to upgrade the whole system anyway (including CPU, I/O, and GPU).
4. Many purchasers of computers (especially Mac Pros) are companies that never upgrade assets on a capital depreciation schedule. When a solution no longer meets the need it is replaced.
The sum of all this is that the number of people truly and negatively impacted by non-upgradable GPUs is small enough to be offset by the benefits of bringing TB to a much larger group of workstation users so they can realize the benefits of greater choice, better support, and more affordable and timely peripherals as discussed earlier.
Now, here on Mac Rumors, the number of people impacted by non-upgradable GPUs appears high, but these forums are the home to enthusiasts who are the most likely to tinker and upgrade systems so that's expected.