Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Apple Silicon in AI (2023)
- Thread starter senttoschool
- Start date
- Sort by reaction score
Good catch! From the Metal Shading Language Specification:
Is this another case like the ray-tracing API where Apple has built software support before hardware support?
Possibly. Another benefit is symmetry with other processors (CPU, AMX, NPU), which support bfloat16 natively.
Apple seems to have put a lot of effort into improving support for ML frames in macOS.
Tensorflow Metal API is 1.0, but Apple didn't bother to write a proper changelog, as usual. The session on ML frameworks highlights grappler pass optimizations, mixed accuracy and simplified installation.
Although Pytorch MPS backend is still in beta, it is now much faster.
developer.apple.com
Tensorflow Metal API is 1.0, but Apple didn't bother to write a proper changelog, as usual. The session on ML frameworks highlights grappler pass optimizations, mixed accuracy and simplified installation.
Although Pytorch MPS backend is still in beta, it is now much faster.
Optimize machine learning for Metal apps - WWDC23 - Videos - Apple Developer
Discover the latest enhancements to accelerated ML training in Metal. Find out about updates to PyTorch and TensorFlow, and learn about...
Last edited:
Post WWDC, Apple execs (and influencers close to Apple) are saying that Apple Silicon isn't in the AI training game. Go do it in the cloud. Which, I think, is consistent with thoughts on this thread.
However, wrt LLMs, what about needs for inference, fine tuning, or even extending models with plugins - like the retrieval plugin? Is Apple ceding those 'non-cloud' tasks to be best performed on workstations from other vendors?
I ask this having not watched any of this year's WWDC content.
However, wrt LLMs, what about needs for inference, fine tuning, or even extending models with plugins - like the retrieval plugin? Is Apple ceding those 'non-cloud' tasks to be best performed on workstations from other vendors?
I ask this having not watched any of this year's WWDC content.
It makes a ton of sense for Apple to cede the training market. Apple has no advantage there. Nvidia has solutions connecting thousands of CPUs and GPUs together. Apple can't compete.
But I think Apple is serious about inference and they've clearly signaled that they are - both doing inference for their own AI apps and providing patches for inference via Tensorflow/Pytorch.
I basically think that by M5 or M6, Apple Silicon will basically be a giant Neural Engine with a CPU and GPU attached to it. Today, it's the other way around.
WWDC updates to CoreML for StableDiffusion.
| Device | --compute-unit | --attention-implementation | End-to-End Latency (s) | Diffusion Speed (iter/s) |
|---|---|---|---|---|
| iPhone 12 Mini | CPU_AND_NE | SPLIT_EINSUM_V2 | 20 | 1.3 |
| iPhone 12 Pro Max | CPU_AND_NE | SPLIT_EINSUM_V2 | 17 | 1.4 |
| iPhone 13 | CPU_AND_NE | SPLIT_EINSUM_V2 | 15 | 1.7 |
| iPhone 13 Pro Max | CPU_AND_NE | SPLIT_EINSUM_V2 | 12 | 1.8 |
| iPhone 14 | CPU_AND_NE | SPLIT_EINSUM_V2 | 13 | 1.8 |
| iPhone 14 Pro Max | CPU_AND_NE | SPLIT_EINSUM_V2 | 9 | 2.3 |
| iPad Pro (M1) | CPU_AND_NE | SPLIT_EINSUM_V2 | 11 | 2.1 |
| iPad Pro (M2) | CPU_AND_NE | SPLIT_EINSUM_V2 | 8 | 2.9 |
| Mac Studio (M1 Ultra) | CPU_AND_GPU | ORIGINAL | 4 | 6.3 |
| Mac Studio (M2 Ultra) | CPU_AND_GPU | ORIGINAL | 3 | 7.6 |
Apple M2 Studio Learns Nuke Machine Learning
The new Nuke with Apple Silicon native CopyCat shows why the new Apple M2 Ultra is a great machine learning VFX tool.
www.fxguide.com
I think Apple means exactly what they said – they're not in the "starting from scratch, using 1000 GPU's" training game. That does not mean they're not interested in the examples you gave like fine tuning.
For example:
- They're using an LLM for the keyboard (and various other things). This will presumably be fined tuned as you type to match your particular language usage.
- They're offering personalized synthetic voices. Right now these are low-ish quality, and intended for people who have difficulty speaking. But at some point this will probably change.
Basically use common sense! If a task is being done on a rack of H-100s, it's not a task Apple thinks should (for now...) be done on a Mac. Otherwise...
Indeed, it's too early to say what will happen in this field. We'll see if the multiple powerful gpus stays the main way to train, or if another way is found. I also believe Apple hasn't given up on the training game, briefly mentioned in the WWDC Keynote:
11:30: “...And M2 Ultra can support an enormous 192 GB of unified memory, which is 50% more than M1 Ultra, enabling it to do things other chips just can’t do. For example, in a single system, it can train massive ML workloads, like large transformer models..."
