I have something new to comment, Apple even has on the cards an modular approach based on 32 line's pcie5 Apu modules on reworked MXP modules (that 7,1 double pcie slots for Dial GPU), each slot consisting on an m2 Ultra or Extreme with everything on board except storage , you could add UpTo 2 of these Ultra Extreme MXP modules alongside 2 or 3 pcie5 x16 slots so the Mac Pro modular could top 80 cores, and be competitive with dual RTX4090 or dual A6000 workstations and include UpTo 768gb of RAM, i know it was discussed but Apple just opted to build a more tradicional Mac Pro with m2 extreme and UpTo 16 ram slots, but I won't be surprised if said ultra MXP modular Mac Pro is the One introduced at WWDC, a thing it's safe it won't disappoint, and offer somehow regular upgrade path on the same chassis, but I still haven't my doubts.
What you describe was something I’ve been speculating about for the last two years. This is the solution I’d ultimately like to see, although it’s not clear to me how the programming model would need to change to make this possible.