Well, the AMX hardware is highly specialized, performs only a subset of operations (mostly matmul stuff), requires specific memory layout etc. GPU is more flexible.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Apple Silicon deep learning performance
- Thread starter gl3lan
- Start date
- Sort by reaction score
Well, the AMX hardware is highly specialized, performs only a subset of operations (mostly matmul stuff), requires specific memory layout etc. GPU is more flexible.
Do you mean this article by Erik Engheim? https://medium.com/swlh/apples-m1-secret-coprocessor-6599492fc1e1
This is a comparison between Apple's M1 matmul performance and ARMv8.6’s standard NEON instructions
Welcome to AMD
AMD delivers leadership high-performance and adaptive computing solutions to advance data center AI, AI PCs, intelligent edge devices, gaming, & beyond.
nod.ai
Welcome to AMD
AMD delivers leadership high-performance and adaptive computing solutions to advance data center AI, AI PCs, intelligent edge devices, gaming, & beyond.
Yes...that article was informative on explaining things a bit...I'm not sure I'm so excited about this new architecture if Apple's going to keep things under lock n key.
Interesting
But in that case, how is that different from Intel's CPU who has instruction optimized for matrix multiplication (Intel even proposes some MKL TensorFlow version for best performance)?
In the end, matrix multiplication is at the core of DL. It would be surprising not to have this optimized in the AME
Edit: the above article helps to understand the interaction between the CPU/AME and AMX
It's still not clear to me how it's different from x86 chips that also have matrix multiplication accelerated hardware
Last edited:
It’s not. Apple AMX is actually fairly similar in design to Intel AMX as far as I can tell. The main difference is that Apple has been shipping matrix coprocessors on all of its products since 2019. Intels stuff will only ship on server class CPUs next year.
These are simply optimized for different usage scenarios.
NPU (AME) seems to be strictly for inference, supports only a subset of models and is extremely power efficient. Of course it has to consist of matrix multiplication units, but they must some sort of fixed function stuff with only limited functionality.
AMX units are more programmable but consume more energy. Also, they are obviously designed to be used in tandem with CPU workflows, as they are driven directly by the CPU instruction.
Finally, GPU is there if you need the most flexibility.
Hi. I tried a training task of image segmentation using TensorFlow/Keras on GPUs, Apple M1 and nVidia Quadro RTX6000. I also made an estimation of M1 Max (I don't have an M1 Max machine).
RTX6000 is 20-times faster than M1(not Max or Pro) SoC, when Automatic Mixed Precision is enabled in RTX...
I posted the various results in Medium.
Training Speed of TensorFlow in macOS Monterey
GPU training in M1 SoC comparing with results in Quadro RTX6000 and estimation in M1 Max SoC
towardsdatascience.com
It seems Tensorflow's performance on M1 GPU improves when the batch size is slightly larger than usual.
@gl3lan Can you check if performance improves when the batch size is 512 or 1024?
Hello gl3lan,
Thank you so much for creating this thread, actually it's the information i've been looking for for a while.
My question is that i got the Macbook Pro 16 inch, 16GB of RAM, 512GB SSD, 10 Core CPU and 16 Core GPU.
Do you think it's more than enough for machine learning developer aor should i upgrade RAM, processor and SSD.
Your answer is very important to me since i can still return the one i got and make a hardware upgrade.
Thank you so much again!
Honestly depends on your use cases, the type of ML (deep or not ?), dataset sizes... For example I happen to work with datasets of hundreds of Gb, but the training is usually done on a server/cloud instance/workstation, where the data is located. I think the machine is sufficient to design and test ML models. For example, you only need to run a DL model on tiny batch sizes/random inputs when developing (small amount of RAM required). Once it is coded you perform training on more capable hardware. Should you require more compute, you are better off with a cloud instance. IMHO upgrading ram or gpu is not worth it considering the cost/performance gain (you won’t train a model for days on a laptop anyway if you can train 10 times faster on a cloud instance). Upgrading SSD might have more sense if you have to perform data analysis locally on huge datasets.
I actually went for the 10/16/1TB version. This is a personal machine but I will sometimes use it for work-related personal projects. My work consists in deep-learning based applied research.
I might invest more in a future MBP/Mac Pro revision, but for me the chip is still WIP regarding machine learning (especially considering it is not yet supported by PyTorch).
Thank you so much,
You answer is more than appreciated. I was really confused and frustrated between the base model or upgrading. But after your answer i'll be keeping the base model. I'm pretty sure now that Macbook pro M1Pro 16 inch, 16gb RAM would be more than enough.
You saved me money and stress.
Thank you again!
Jordan Harrod has compared the performance of M1 Pro and M1 Max chip for machine learning tasks.
No choice under the circumstances unless ARM adopts a similar or better instruction set. ARM v9 does indeed introduce such a thing so we’ll see if Apple adopts it or maybe it will be similar to what Apple was already doing. But right now accessible through an API is the only way.
Last edited:
Differences are at a technical level.
- AMX instructions look like normal ARM instructions and are embedded in the standard ARM instruction stream, executed in a core's load/store unit.
- the instructions are sent to the L2 of the cluster the core belongs to (P, or E) and are executed by the AMX unit for tha t cluster, which can be considered as associated with the L2. Essentially it uses the L2 as its "cache" and has low latency access to it.
- AMX instructions provide at least double, float, and fp16 instructions.
Intel AMX seems to provide only int8, int16 (I think) and bf16 instructions. It's unclear to me how tightly Intel AMX is coupled to L2 (or L3) (ie low latency load/store) or to the instruction stream (ie low latency dispatch of instructions rather than a whole drama of "creating a code package to mail to an accelerator then tell it to execute that")
Apple provides a MATRIX accelerator, good for a variety of FP work.
Intel provides an AI accelerator that does the job for INFERENCE and may be good enough for most training. (Training almost always requires FP, but bf16 may be good enough.) But Intel's solution won't help if, for example, you are more interested in solving PDEs or physics/engineering rather than just AI.
I'm honestly not sure quite why Apple gave SUCH A POWERFUL generic HPC tool. But, as someone who uses a lot of Mathematica, I'm grateful! (Or will be, once a fully-optimized version of Mathematica is released, not the current bare-bones "should be called beta" release.)
I get the impression that some scientific open source projects (Numpy, Pytorch...) can't develop for Apple Silicon GPU as fast as they would like to because Github doesn't provide that platform.
Is it true? Would Apple change their MacOs license for servers? Would an Apple Mini Pro change the situation?
Is it true? Would Apple change their MacOs license for servers? Would an Apple Mini Pro change the situation?
Hi Maynard, great to have someone with your knowledge level comment on this! Regarding AMX, I am not quite sure where Apple is going... We have one AMX unit per cluster of 4 P-cores, so if I read Dougall's notes correctly, that would translate to around ~ 3 TFLOPS FP32 throughput with ideal utilization on multiple cores. This is of course great for general vector algebra tasks, but not nearly enough to challenge the deep learning performance of larger GPUs.
I have the feeling that Apple still has to figure these things out properly. Their hybrid solution seems to lack a bit of balance here. Right now, for large matrix-related tasks, it makes sense to use AMX on the base M1, but you might be better off using the GPU on the M1 Max. This obviously creates a weird situation for the developers... besides, it appears that Apple's new tensorflow plugin does not use AMX at all, which again makes me question the future of these units.
Not sure if the following twitter thread has been mentioned before. Someone posted a 2.7 TFLOPs result using
gemm-benchmark, which seems to be in the ballpark.There's also a die shot annotation that looks pretty thorough (although it's missing all of the PCIe and serial stuff at the top, and there are two blocks above and below the left hand SLC that I'm curious about). Anyway, the AMX blocks marked out in the die shots are so small compared to the GPU cores, while their sgemm TFLOPS/block is ~1.4 vs 0.3 with the GPU. If the annotation is accurate, I wonder why there wouldn't be more focus on AMX.
Edit: Oops, yeah, the twitter thread was posted earlier in this thread. Having trouble keeping threads straight ?♂️
Last edited:
Where do you think Apple wants to sell these machines?
The fact that AI/ML is the hot new thing (and so the only thing we ever hear talked about by journalists and pundits) doesn't change the fact that vastly more engineers and scientists work in "traditional" hard sciences than in AI/ML. And AMX is a very nice performance boost for those folk...
Read what I said. AMX is a MATRIX accelerator. Do you want a computer than can perform matrix math fast (ie are you the sort of person who would, twenty years ago, have wanted to own a "workstation")? Then Apple Macs are a much nicer machine for you than the competition (x86 today, maybe ARM tomorrow). AMX can handle ENGINEERING/PHYSICS matrix math -- doubles and floats.
If you have something that works well on the GPUs, great. But latency computing is different from throughput computing. With AMX Apple is covering all bases, rather than forcing matrix computing into the throughput mold even when your problem is a latency problem:
Ultimately I think you are hostage to the monic fallacy (everything must have a SINGLE solution) that has crippled so much of computing. This viewpoint is what drove MS over a cliff ("let's use the same UI for everything from phones to servers") and Intel ("let's force x86 even into circumstances, like phons, where it makes no sense"). You see it in the people who insist that the iPad be turned into a Mac, or that the aWatch be able to do everything without an iPhone.
Apple is not immune to this diseases, just mostly better at fighting it off than most companies.
Who cares if there is an AMX block on a chip or not? Transistors are cheap! Providing an AMX block (and hiding it behind Accelerate APIs) means that it can be used where appropriate (high precision math, low latency requirements) and not used where appropriate (low precision math, throughput requirements).
I agree that the fp16 support in AMX is somewhat weird and I suspect at some point that may be tossed; it doesn't seem relevant to the primary (hard sciences, low latency) use cases I see for AMX. But let's see... Maybe it will turn out that there are, eg, ways to solve elliptical PDEs (think something like Laplace or Poisson) that run the first few rounds of relaxation in fp16 before switching to fp32 then ultimately fp64, and by cascading accuracy this way you can solve the PDE 30% faster than starting directly at fp64?
How can we know whether scientific programs/libraries like Numpy, Julia or Matlab use Apple's AMX to speed up matrix multiplications?
Numpy uses Apple’s Accelerate.framework, which uses the AMX2. Not sure about the others, but it shouldn’t be hard to look up.
Last edited:
Not sure if it guarantees AMX use, but if you run otool -l ”path to application” it’ll tell you what frameworks are linked.