Apple Silicon in AI (2023)

[senttoschool]

For those working in the field of AI (training, inference, research), what is the state of Apple Silicon in 2023?

How are frameworks such as Pytorch and Tensorflow support for Apple Silicon coming along? Are any major optimizations expected in 2023?

If you don't rent an Nvidia GPU in the cloud, are you completely hampered?

 

[Xiao_Xi]

How are frameworks such as Pytorch and Tensorflow support for Apple Silicon coming along?

Not so great because Apple seems to care more about inference (CoreML) than training (Pytorch/Tensorflow).

PyTorch is officially in beta and Tensorflow should be too because after two years it still has lots of problems [1] and poor documentation [2].

[1] https://developer.apple.com/forums/tags/tensorflow-metal

[2] Apple hasn't added the info of v2.10 and v2.11 although it released v2.10 last September and v2.11 last December.

Tensorflow Plugin - Metal - Apple Developer

Accelerate the training of machine learning models with TensorFlow right on your Mac.

developer.apple.com

 

[teagls]

Trying to do any accelerated AI training with PyTorch/Tensorflow is a waste of time on MacOS. If you want simpler stuff (unsupervised learning, SVM, RND Forest, etc.) scikit-learn is usable on MacOS. Cuda is still king. You are best off buying a low end Nvidia GPU and building a linux machine. It will run circles around any Mac for deep learning.

 

[Pedro1223]

I'm using TensorFlow optimized for mac m1. As far as I have just seen a couple of days it is incredibly fast and convenient. The image should give you an idea.
EDIT: I forgot this is in low power mode only with battery. Plugged in is 2/3s faster.

 

[Kimmo]

Interesting article on Nvidia's A100 and its newer, Hopper generation, H100.

Meet the $10,000 Nvidia chip powering the race for A.I.

The $10,000 Nvidia A100has become one of the most critical tools in the artificial intelligence industry,

www.cnbc.com

 

[Xiao_Xi]

Facebook's LLaMA has been adapted to run on Apple Silicon.

GitHub - ggerganov/llama.cpp: LLM inference in C/C++

LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.

github.com

Using LLaMA with M1 Mac

CPU only LLaMA on M1 Mac with Python 3.11

dev.l1x.be

 

[senttoschool]

Much easier to install LlaMa with this: https://cocktailpeanut.github.io/dalai/#/

 

[sunny5]

I'm using WebUI to create images but it's def slow and not really taking advantage from high VRAM spaces.

 

[Xiao_Xi]

PyTorch is slowly improving its correctness, stability and operator coverage on Mac platforms.

MPS backend now includes support for the Top 60 most used ops, along with the most frequently requested operations by the community, bringing coverage to over 300 operators. The major focus of the release was to enable full OpInfo-based forward and gradient mode testing to address silent correctness issues. These changes have resulted in wider adoption of MPS backend by 3rd party networks such as Stable Diffusion, YoloV5, WhisperAI, along with increased coverage for Torchbench networks and Basic tutorials.

PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever

We are excited to announce the release of PyTorch® 2.0 which we highlighted during the PyTorch Conference on 12/2/22! PyTorch 2.0 offers the same eager-mode development and user experience, while fundamentally changing and supercharging how PyTorch operates at compiler level under the hood with...

pytorch.org

 

[Retskrad]

Apple can't even do rudimentary AI right, like autocorrect on the iPhone keyboard. What makes people think they have any chance whatsoever in the cutting edge AI wars that we are in right now? The thing that saves Apple in these times is Johny Srouji and his team who made a wise decision years ago to invest heavily in the Neural Engine in Apple's silicon. Thanks to that investment, Apple can step out of the way and let other companies use their vastly better AI and build on top Apple hardware. That's a win-win for Apple. Let's be honest, nobody cares about Apple's services or their Apps. It's all about their hardware and the ecosystem. Not a single Apple app or service is better than the competition, lol!

 

[PineappleCake]

Not a single Apple app or service is better than the competition, lol!

Very true. Even Final Cut pales in front of Resolve.

Apple Music is meh apart from lossless music. Tidal/YT Music is better for me.

 

[teagls]

Apple can't even do rudimentary AI right, like autocorrect on the iPhone keyboard. What makes people think they have any chance whatsoever in the cutting edge AI wars that we are in right now? The thing that saves Apple in these times is Johny Srouji and his team who made a wise decision years ago to invest heavily in the Neural Engine in Apple's silicon. Thanks to that investment, Apple can step out of the way and let other companies use their vastly better AI and build on top Apple hardware. That's a win-win for Apple. Let's be honest, nobody cares about Apple's services or their Apps. It's all about their hardware and the ecosystem. Not a single Apple app or service is better than the competition, lol!

This is a good point. Apple was always very good at hardware design and software integration. They've never been good at cutting edge or advanced software capabilities. Back in the day their OpenGL drivers were garbage. They never updated their OpenGL spec. Metal was always lacking in features compared to its cross platform equivalent. CoreML is basically a black box. Their model conversion tools to CoreML are garbage and littered with bugs. They have a track record of mediocrity and hamstringing developers. On the other hand, although Nvidia has their faults. They produce very sophisticated and polished software. There is a reason CUDA is king. Until Apple changes their software approach towards developers, AI will always be lackluster and lag behind on their platforms.

 

[leman]

Apple was always very good at hardware design and software integration. They've never been good at cutting edge or advanced software capabilities.

That’s hardly a fair assessment. Their OS-wide search technology, desktop compositor, and file system/backup technology stack are state of the art, for example. They ship many advanced APIs as part of the OS, including an async runtime with full OS integration etc. While Siri and autocorrect could be better, their sound and image services are top notch. Just to mention a few things.

Metal was always lacking in features compared to its cross platform equivalent.

That was only really the case with the first Metal version, which was supposed to be a basic easy to use API. I’d argue that these days it’s mostly the other way around. Vulkan became a convoluted mess for example and has been trying to catch up with Metal.

They have a track record of mediocrity and hamstringing developers.

This is true. Apples weak point is communication and transparency. Their documentation is generally garbage. They start interesting projects but don’t bring them to the end and the feedback channels are pretty much non-existent. There is a lot of work to do in that area.

 

[TechnoMonk]

I have used multiple AI pipelines for months.

Training: A 100 in the cloud, H100 provides more clustering and micro-GPU granularity flexibility. Apple is nonexistent in this space; I wonder if Apple will ever complete this space.

Inference: I want to spend only a little money on cloud-running inferences; this is where I spend most of the time after I have trained models ready for action. This is the most frustrating part of my workflow using NVidia 4090 or 3090. Very unstable, need to limit power to 70%, underclock by 20-25%. These run out of memory often; 24 GB is a joke these days. I have 128-256 GB RAM available, which GPU can't use.

I was forced to convert some of my models to COREML, it was a joke a year ago, but Apple finally woke up and has been pretty decent in recent months. My 16 " M1 max MacBook pro sometimes performs better with 64 GB unified memory. In some cases, it's my only option.

Yes, I have an extra step to convert my Models trained on A 100; I had to build a wrapper on the existing code base to support AS. In the end, I have the best of both worlds. I could spend 5K on A5000, but it's like jets taking off in my backyard, not to mention the power draw and stability issues.

 

[Ethosik]

Very true. Even Final Cut pales in front of Resolve.

Apple Music is meh apart from lossless music. Tidal/YT Music is better for me.

Davinci is not working well for me. But FCP is working great. It depends on the user.

 

[TechnoMonk]

Very true. Even Final Cut pales in front of Resolve.

Apple Music is meh apart from lossless music. Tidal/YT Music is better for me.

Davinci is not working well for me. But FCP is working great. It depends on the user.

They both have good and bad, I find FCP more intuitive and easier than Resolve for my needs.

 

[Xiao_Xi]

Honestly, software for Apple hardware is not that far behind in the AI field. With extra effort from Apple engineers, the Tensorflow and Pytorch Metal backend can be as good as Nvidia's backend.

 

[PineappleCake]

FCP more intuitive and easier than Resolve for my needs.

I find the opposite to be true. Apples UI for me is confusing.

 

[PineappleCake]

Davinci is not working well for me. But FCP is working great. It depends on the user.

That'd good to hear. Glad it works for you.

 

[TechnoMonk]

Honestly, software for Apple hardware is not that far behind in the AI field. With extra effort from Apple engineers, the Tensorflow and Pytorch Metal backend can be as good as Nvidia's backend.

Apple still needs to add speciliazed RT cores in HW. Problem is not so much on PyTorch or TF backend but other depended open source libraries. Apple needs to invest in bridging some of those gaps.

 

[senttoschool]

Inference: I want to spend only a little money on cloud-running inferences; this is where I spend most of the time after I have trained models ready for action. This is the most frustrating part of my workflow using NVidia 4090 or 3090. Very unstable, need to limit power to 70%, underclock by 20-25%. These run out of memory often; 24 GB is a joke these days. I have 128-256 GB RAM available, which GPU can't use.

I was forced to convert some of my models to COREML, it was a joke a year ago, but Apple finally woke up and has been pretty decent in recent months. My 16 " M1 max MacBook pro sometimes performs better with 64 GB unified memory. In some cases, it's my only option.

Am I understanding it right that your 64GB AS does inference better than a 24GB RTX?

 

[sunny5]

Am I understanding it right that your 64GB AS does inference better than a 24GB RTX?

You gotta think about the memory bandwidth which is faster than M1 Ultra's 800 gb/s. Having 64GB unified memory doesn't really mean too much especially for 3D and AI due to lack of optimizations.

 

[Xiao_Xi]

Problem is not so much on PyTorch or TF backend but other depended open source libraries.

Which ones do you think are the most important?

 

[teagls]

I have used multiple AI pipelines for months.

Training: A 100 in the cloud, H100 provides more clustering and micro-GPU granularity flexibility. Apple is nonexistent in this space; I wonder if Apple will ever complete this space.

Inference: I want to spend only a little money on cloud-running inferences; this is where I spend most of the time after I have trained models ready for action. This is the most frustrating part of my workflow using NVidia 4090 or 3090. Very unstable, need to limit power to 70%, underclock by 20-25%. These run out of memory often; 24 GB is a joke these days. I have 128-256 GB RAM available, which GPU can't use.

I was forced to convert some of my models to COREML, it was a joke a year ago, but Apple finally woke up and has been pretty decent in recent months. My 16 " M1 max MacBook pro sometimes performs better with 64 GB unified memory. In some cases, it's my only option.

Yes, I have an extra step to convert my Models trained on A 100; I had to build a wrapper on the existing code base to support AS. In the end, I have the best of both worlds. I could spend 5K on A5000, but it's like jets taking off in my backyard, not to mention the power draw and stability issues.

Inference on Nvidia hardware is very good... they have tons of optimizations for it and even more depending on which framework like PyTorch/tensorflow you use. I have had zero issues with inference. It's very solid.

Conversion to CoreML is still a nightmare. I dread it the most. It is the most painful god forsaken part of dealing with AI in Apple's ecosystem. The python package Coremltools developed by Apple is riddled with bugs, missing features, terrible documentation. I have submitted countless bugs and they go unanswered. Take for instance this INCREDIBLY basic tensorflow function unravel_index. This bug was submitted almost 2 years ago and was never addressed!!!! https://github.com/apple/coremltools/issues/1195

The irony is Apple internally uses Nvidia and PyTorch for their own AI workflows. They'd never use their own ecosystem.

 

[TechnoMonk]

Am I understanding it right that your 64GB AS does inference better than a 24GB RTX?

Sometimes. Its more stable and only option for me when running models which need more than 24 GB of RAM. NVidia should start offering at least 40-48 GB RAM.

Which ones do you think are the most important?

All are important, chances are in a complex pipeline you are using combination, not just one or the other.