The Ultra itself is a multi-thousand dollar upgrade, so the Extreme would almost have to be...
Teasing out the cost of the M2 Ultra by itself, it seems to be between $1300 and $1400 as an upgrade. The Mac Studio costs $2000 extra with the smaller-GPU version of the Ultra, but it also comes with a $400 RAM upgrade, a $200 storage upgrade and two Thunderbolt ports replacing USB. The Thunderbolt ports are not for sale separately, but are probably not terribly valuable on a machine that already has four (I'm putting them between $0 and $100, but that is a complete guess).
The full-power M3 Ultra is very expensive ($1000 extra over the low-GPU version). That is interesting because the difference on a Max is only $200, suggesting that an Ultra should be around $400. That's either profiteering, extra yield issues or both.
If we interpolate for the M3 line, there's not a ton of evidence, but it's not going to be cheap. The base M3 Max is $200 more expensive than the M2 Max (minus a little for the difference between 32 and 36 GB of RAM - would that be a $50 upgrade if it existed???).
I'd guess that base M3 Max to base M3 Ultra will be a $1500 -$1800 upgrade - a little more than the $1300-$1400 in the M2 generation.
What's much costlier in the M3 generation than the M2 generation is the upgrade from a base Max to a loaded Max ($500 instead of $200). The differences are much more important in the new generation - it's a bigger change in GPU cores, adds two P-cores to the CPU, and changes the memory bus. I'd suspect the difference from base Ultra to loaded Ultra will be somewhere between $1500 and $2000 (the old $1000 difference would be in line with what the Max costs, but it hasn't been - there's been a significant premium for upgrading an Ultra as opposed to upgrading two Maxes). If that is a legitimate yield issue, an Ultra isn't "just two Maxes from anywhere", and a loaded M3 Ultra (let alone a hypothetical M3 Extreme) could be very difficult indeed to manufacture - but it's possible that Apple is just profiteering.
We're already looking at a $3000-$4000 total upgrade from a base M3 Max to a loaded M3 Ultra, and the upgrade to Extreme would be AT least that much again, plus the cost of a base M3 Max (somewhere around $600). If there are yield issues, it could be more. At a guess, it will be about $8000 to go from a base M3 Max to a loaded M3 Extreme (maybe $5000 if you settle for a 40 P-core with 120 graphics cores, instead of 48 P-cores and 160 graphics cores).
The Extreme would probably only fit in a Mac Pro, which comes with a base Ultra for $6999. How about $9999 for a base Extreme and $12,499 for a loaded Extreme (before RAM and storage options)? It might even be a little more because of obligatory RAM upgrades...
Oddly, that's a deal... The Extreme should compete pretty easily with the really big Xeon and Threadripper Pro chips (which are $10,000 for the chip alone - and that's before your system builder marks it up - they cost $12,000-$13000 just for the chip if you get them installed) The M3 Extreme may not be as fast in weirdly parallel workloads, since it's a 56 or 64 core chip (and 16 of those are e-cores), while the Threadripper Pros are as much as 96 cores, all "big" On the other hand, the Threadripper cores are quite a bit slower than Apple's, and even if they are 2/3 the speed, trying to schedule them will eat up the difference in theoretical power (with the possible exception of very parallel jobs, like lots of small database queries). The Threadripper Pros and big Xeons are server chips adapted to workstations, and they're optimized for server work.
A Threadripper Pro 7995 WX machine is going to be significantly more expensive than the Mac Pro - around $20,000 from the first few vendors to hit the market for a pre-built machine with Apple's presumed pathetic base specs (128 GB of RAM and 2 TB of storage). That includes a single RTX 4090 GPU, which may or may not be as fast as the Mac's 160-core GPU. The RAM upgrade to 512 GB (Apple's probable top spec) is somewhat cheaper than Apple charges - ~$3000 on the Threadripper, and I bet Apple will try to get $6000 (judging from the usual RAM prices on Macs), which eats up some of the difference, but not all of it. Yes, you could get the Threadripper to 512 GB for significantly less than your system builder would charge by upgrading the RAM yourself, but, realistically, owners of $20,000 workstations don't do that - those machines are under service contracts. Apple will also overcharge for storage - but there is probably no reason to let them on a tower with multiple PCIe and Thunderbolt storage options. The only catch might be things the operating system won't let you put on another drive - and I don't know how MacOS will handle other internal drives, since the Mac Pro is the ONLY Mac that can have internal drives other than the boot drive, and I don't have one around.
The one real advantage of the Threadripper is that it can scale beyond where the Mac Pro can. Threadripper Pro CPUs are limited to single-CPU installations - but closely related Epyc chips aren't. Similarly, Xeons that support up to 8 CPUs per system are available. You'll pay even MORE per CPU for those, but if you want hundreds of CPU cores in one box, you can have them. Similarly, two and even four GPUs per system are possible, and you can even use eight by using much more expensive workstation GPUs (you'll pay more than twice as much per FLOPS, but you can use more per system).
The sky is literally the limit - a modern supercomputer is made up of a LOT of AMD or Intel CPUs with AMD or Nvidia GPUs, and if you call up Cray and ask "how many can you do in a box", they'll reply "how many are you willing to pay for?" and/or "how much electricity and cooling do you have?" The only way to do that with M-series chips is to cluster a bunch of individual computers. I wouldn't be terribly surprised if we see some M3 Ultra (etc.) clusters - several "supercomputers made out of Macs" were built in the early G5 era, when the PowerPC G5 significantly outperformed Intel chips for a couple of years, but wasn't available in dedicated supercomputing boards. The performance per watt of the M-series is enough better than the competition that I wouldn't be surprised if some university's computer science department is contemplating a big cluster (homemade supercomputers are almost always academic).