I believe Apple's cooling problems are predominately due to Intel and AMD so badly missing their TDP targets they gave Apple (and other OEMs), who designed their chassis around chips that were expected to run a fair bit cooler then they did.
It's not so much about "missing" TDP targets as Intel's means of measuring TDP doesn't match how their CPUs work in a post Turbo Boost world. When your CPU has a feature to self-overclock, and your TDP is measured with it disabled or not active, it's a less useful metric when designing systems. I wonder if Intel realized that consumption under boost is varied enough that it doesn't exactly make sense to report a single TDP when each copy of the silicon will behave slightly differently under boost.
But at the same time, it's not like Apple and other OEMs aren't aware of this behavior. They have been seeing it in their engineering labs for years at this point. And Apple could configure their systems to obey the TDP when boosting, at the cost of folks complaining, if they really wanted to.
That said, I am pretty impressed that the 2018 Mac Mini is able to handle dissipating ~80-90W in the same case as the 2014, which used chips with less than half the TDP. And still be quiet while doing it.
If Apple could have made this work and reduced their reliance on Intel, they would have done. Even if it meant stacking loads of cores/processors together.
Something fundamental is lacking on the performance side.
Not necessarily. As others have pointed out, things like PCIe would need to be supported, and generally you need a lot better I/O support. Something that an AX SOC doesn't care about in iPhones or iPads. If you want to support Thunderbolt, you need PCIe lanes available. So even a laptop would need access to extra I/O if you wanted to hook up a Titan Ridge controller. So there's some "despecialization" that has to happen before an ARM chip could replace Intel/AMD in laptops or desktops.
Sometimes it isn't even about scaling the design, but the engineering resources required to do it, and how far to carry prototyping/etc. Apple is in a different world today than 2005. AMD and Intel both have a market they are interested in serving, and Apple is part of that market. It's not like when Motorola was wondering if they should care about non-embedded CPUs, and IBM wondering if they should care about non-server CPUs. But Apple's also different in that in 2005, Mac was their primary business by revenue with iPod a close second. Now, the iPhone is, in a market where CPUs is heavily driven by Qualcomm and Samsung. So I'd say the business case for custom SOCs is stronger for the iPhone than it is for the Mac.
I'd be surprised if Apple didn't have prototypes. iOS and macOS are already similar enough that you could do it and get a rough idea what the experience is like, by standing up WindowServer and AppKit on the iPad Pro. I'd also be surprised if Apple didn't squeeze the Intel and AMD relationships as much as possible before making the leap themselves in the coming years, to avoid a ton of investment in what is effectively commodity hardware (yet important commodity hardware to Apple because of macOS).