MP All Models Apple sleeping on the mac pro is comically bad timing

[Charlie Bonesx]

I don't think an old Mac Pro with 1.5TB memory is going to be better for an AI than a 512GB Mac Studio, even if it has 3 times as much memory.

It’s absolutely not

We’re talking about a nearly decade old machine here

You can jam as many old amd gpus and as much ram as you want in to it

Unified memory aside, the giant old cpu is a massive bottleneck

Even a base m4 mini is going to run circles around it

Sure you get more total pcie pass through with the Xeon, but the bottleneck is still there

 

[Charlie Bonesx]

post-purchase hardware reconfigurability, and in-service field upgrades.

In reality though, in most enterprise contexts, isn’t there more to be gained from updating chipsets then adding ram and storage later on?

 

[teh_hunterer]

In reality though, in most enterprise contexts, isn’t there more to be gained from updating chipsets then adding ram and storage later on?

Not only that, but enterprises don't buy less than they need and then add more RAM and storage over time. They buy what they need when they need it and just replace it when it's time to replace it.

This idea of:

post-purchase hardware reconfigurability, and in-service field upgrades / repairs.

in the context of PCs provided to employees at the enterprise level is mostly just a fantasy. Some Redditor enthusiast might enjoy swapping around parts in their desktop tower while they LARP as a professional, but actual professional use cases involving this stuff involves the purchase of the thing that is actually required.

Any time you have a technician having to go to the location of a PC, open it up, change things around, you're spending so much money on that labor that you would have saved money just buying the specced up version in the first place. If you're an enthusiast at home, by all means do all that free labor for yourself, but don't confuse Reddit fantasy with enterprise reality.

 

[Audentia]

As the video shows - even a single Mac Studio is a decent machine for local LLM because of the advantage of having a relatively large amount of Unified RAM directly shared between CPU/GPU/NPU vs. limited VRAM (which has to be loaded via main RAM) on a dGPU card.


An Apple Silicon Mac Pro with a competitive level of bandwidth+lanes for PCIe based GPUs or NPUs isn't possible (without Apple designing a new SoC die just for the Mac Pro) and doesn't exist... and, even if it did, would be reliant on the same NVIDIA or AMD dGPUs that can be used in any generic Xeon/Ryzen box and lack the unified RAM advantage which lets a Mx processor with integrated GPU punch above its weight.

The Mac Pro may look like it has plenty of PCIe slots, but the M2/M3 Ultra only has 32 lanes of PCIe4 of which only 22 are available for the PCIe slots (with various constraints on how they can be allocated to slots). Current Xeons and Threadrippers have 128+ lanes of PCIe5. AFAIK many LLM tools run in Linux just as well as MacOS/Unix, and CPU power consumption is irrelevant on a personal workstation stuffed with NVIDIA space heaters, so if you want a GPU-based LLM platform a Xeon or Threadripper is probably the tool for the job.

Want a Mac Pro cluster? - you'd have to use Thunderbolt, same as the Studio.

Agreed, I think the missing thing or what's being unsaid from the mac pro advocates: we (or at least I) assumed Apple would release special versions of the high end Apple Silicon chips suitable for scaling and expansion in pcie slots when they first announced the Apple Silicon Mac Pro. Basically like you buy an M2 Ultra Mac Pro, but then you can add a M4 Max via pcie 5 board and the system expands like what they did with the Ultra fusion chips anyway. Software abstracts it and makes it a user friendly power user experience.

Now as you said it would need to be Apple Silicon that doesn't actually exist with more PCIe lanes or maybe even PCIe wasnt enough and they needed a custom connector like NVLink. This would then make the chassis costs worth it, and a machine you could own and upgrade and invest in over a long time. Apple could sell various boards like Framework and still make money with decent markup. You could say maybe it's too niche and not worth the cost for apple to invest in this ecosystem, but the idea from the power user is ok you could have a machine several times more powerful than a Mac Studio.

This seems to be exactly what Nvidia DGX Station https://www.nvidia.com/en-us/products/workstations/dgx-station/ is doing, so with the right willpower its obviously technically feasible, but sadly Apple's execs did not have this vision and now Nvidia may eat their lunch in a whole new category this fall. Essentially RTX Spark competes with MacBook Pro and Mac Studio Apple Silicon, DGX Station has no competition, but it could have been a Mac Pro and specialized high end Apple Silicon. This is why I completely agree with the OP's presuppositions. I think a lot of people would prefer to host powerful models on macOS rather than windows, it's hugely disappointing for Apple to concede this market and hope John changes things.

 

[leman]

Now as you said it would need to be Apple Silicon that doesn't actually exist with more PCIe lanes or maybe even PCIe wasnt enough and they needed a custom connector like NVLink. This would then make the chassis costs worth it, and a machine you could own and upgrade and invest in over a long time. Apple could sell various boards like Framework and still make money with decent markup. You could say maybe it's too niche and not worth the cost for apple to invest in this ecosystem, but the idea from the power user is ok you could have a machine several times more powerful than a Mac Studio.

It has already been pointed out that a price for such a hypothetical platform would likely far surpass what the enthusiasts would be willing to pay, so I am not going to argue this point again.

What I do want to point out is that Apple has offered an alternative path towards scalability (one they are themselves using for their custom AI servers), and that is the ability to connect multiple Macs and distribute work between them. It's not seamless, and you need to develop software to take care of such setups, but if you have the need, it could be a viable solution for running large ML models locally. It would be great if Apple included faster connectivity options with the future Studio models.

This seems to be exactly what Nvidia DGX Station https://www.nvidia.com/en-us/products/workstations/dgx-station/ is doing, so with the right willpower its obviously technically feasible, but sadly Apple's execs did not have this vision and now Nvidia may eat their lunch in a whole new category this fall.

Unfortunately, there is a lot of misunderstanding surrounding the DGX Station. That is a supercomputer chip packed into a portable chassis. These machines cost close to $100K, they are not modular and do not come with upgradeable components. They offer only minimal PCI-e connectivity, mainly for a traditional GPU since the supercomputer chip lacks any graphics processing capabilities. So no, DGX Station is _not_ an example for a flexible personal workstation you seem to have in mind, it's quite the opposite.

 

[teh_hunterer]

we (or at least I) assumed Apple would release special versions of the high end Apple Silicon chips suitable for scaling and expansion in pcie slots when they first announced the Apple Silicon Mac Pro. Basically like you buy an M2 Ultra Mac Pro, but then you can add a M4 Max via pcie 5 board and the system expands like what they did with the Ultra fusion chips anyway. Software abstracts it and makes it a user friendly power user experience.

That isn't feasible at all. The whole point is to have the CPU and GPU and memory on the same chip, because the reduced physical distance between everything is what makes it more efficient and powerful.

Introducing PCIe boards throws all that away for nothing. Nobody is ever going to build that. Physics doesn't agree with it.

This seems to be exactly what Nvidia DGX Station https://www.nvidia.com/en-us/products/workstations/dgx-station/.

No? It doesn't work like what you described at all. Nor would it, because that would be a very ineffective and wasteful way to build that tech. As has been pointed out that stuff costs something in the $100k range.

Look I get that the idea of having this desktop tower that you can keep adding stuff to modularly seems neat, but computing is permanently moving away from it.