Apple Intelligence on M series future?

[macman4789]

Hi,

Looking ahead to the use of AI in Macs, I’m looking at the ‘on paper’ differences in TOPS and the significant jump in the M4. From memory, M1 is around 11 TOPS, M2 14-15, M3 including Pro/Max 18 TOPS and then M4 is around double at 36 TOPS I think.

Clearly, Apple have massively ramped up the AI abilities on paper with the M4 but what realistically does this mean long term? The cynical side of me is thinking after the first wave of AI is fully implemented and all of the currently announced features are here, that Apple will then limit any new features to M4 chips and above based on the big jump in TOPS and that will be their ‘justification’ for it. Which would be unacceptable based on how they promote the M3 chips as AI ready just last year.

What are your thoughts?

 

[coffeemilktea]

I think Apple is so far behind on AI, that by the time there's any new and exciting features worth looking forward to, we'll be shopping for M6 and M7 Macs.

 

[novagamer]

Hi,

Looking ahead to the use of AI in Macs, I’m looking at the ‘on paper’ differences in TOPS and the significant jump in the M4. From memory, M1 is around 11 TOPS, M2 14-15, M3 including Pro/Max 18 TOPS and then M4 is around double at 36 TOPS I think.

Clearly, Apple have massively ramped up the AI abilities on paper with the M4 but what realistically does this mean long term? The cynical side of me is thinking after the first wave of AI is fully implemented and all of the currently announced features are here, that Apple will then limit any new features to M4 chips and above based on the big jump in TOPS and that will be their ‘justification’ for it. Which would be unacceptable based on how they promote the M3 chips as AI ready just last year.

What are your thoughts?

There is no large jump in TOPS with M4. It’s less than 10% faster than M3, they’re just using different metrics.

I expect we won’t see massive gains in AI/ML for 2 years in the M series. THe “Hidra” may get one next summer though, I wasn’t aware it was a desktop focused architecture until this week.

M5 MBP will be a redesign and sell very well due to that, maybe including OLED, maybe not. Huge jump in performance probably reserved for the following year, they need their highest selling product to be compelling in some key way every iteration, and they got it this time with the M4 Pro being significantly better than the M3 Pro. Antiglare, bandwdith etc. as well as a huge single core bump gain gave them enough to do so this time.

We won’t see huge TOPS gains unless Apple starts giving us more accelerators or redesigned cores and I have a feeling after learning about Hidra that it won’t be until M6 when some of that development folds back into the M series.

 

[komuh]

The number of TOPS/TFlops is still quite limited, indicating a significant amount of growth potential before we truly achieve “AI” readiness. Unfortunately, ANE is severely constrained, and most applications still run on GPUs. However, there’s a glimmer of hope that we might be able to develop “shaders” for ANE in the next few years.

 

[WarmWinterHat]

The number of TOPS/TFlops is still quite limited, indicating a significant amount of growth potential before we truly achieve “AI” readiness. Unfortunately, ANE is severely constrained, and most applications still run on GPUs. However, there’s a glimmer of hope that we might be able to develop “shaders” for ANE in the next few years.

I have yet to see any clear explanation of what TOPS means to anything, other than just being a bigger and bigger number. Is there ANY real-world example of the difference between 11 TOPS and 14 TOPS? What I can do, specifically, on one and not the other?

I just end up glossing over anything with TOPS because there seems to be no real-world examples.

 

[leman]

I think Apple is so far behind on AI, that by the time there's any new and exciting features worth looking forward to, we'll be shopping for M6 and M7 Macs.

Can you in into more detail here? Maybe specify what exactly you mean by AI and which models you are interested in?

It seems Apple is doing quite well in the domain of on-device low-power inference. It also has the fastest currently shipping CPU ML accelerators. Of course, it is far behind Nvidia in dedicated ML training, and has no place at all in large model training.

I have yet to see any clear explanation of what TOPS means to anything, other than just being a bigger and bigger number. Is there ANY real-world example of the difference between 11 TOPS and 14 TOPS? What I can do, specifically, on one and not the other?

I just end up glossing over anything with TOPS because there seems to be no real-world examples.

TOPS doesn't seem to mean much in practice, since hardware capabilities and software implementations differ. You can have all TOPS in the world, but it doesn't mean much if you can't use it. Just look at the iPhones. Not very impressive in their TOPS ratings, and yet they outperform pretty much anything in on-device ML tests.

 

[macman4789]

There is no large jump in TOPS with M4. It’s less than 10% faster than M3, they’re just using different metrics.

I expect we won’t see massive gains in AI/ML for 2 years in the M series. THe “Hidra” may get one next summer though, I wasn’t aware it was a desktop focused architecture until this week.

M5 MBP will be a redesign and sell very well due to that, maybe including OLED, maybe not. Huge jump in performance probably reserved for the following year, they need their highest selling product to be compelling in some key way every iteration, and they got it this time with the M4 Pro being significantly better than the M3 Pro. Antiglare, bandwdith etc. as well as a huge single core bump gain gave them enough to do so this time.

We won’t see huge TOPS gains unless Apple starts giving us more accelerators or redesigned cores and I have a feeling after learning about Hidra that it won’t be until M6 when some of that development folds back into the M series.

I know they quote as saying ‘10% faster’ or whatever but yet the number of TOPS between M3 and M4 is 18 to 36 which obviously is a significant jump. Now, I’m no expert at all with this so could anyone explain how there can be such a large difference in numbers yet only 10% increase in speed?

 

[terminator-jq]

This is actually one of the top reasons I am going for the M4 Max rather than catching a good deal on an M3 Max (which are already starting to pop up). As you said, the jump in M4 A.I. processing abilities is significant. That's pretty telling in itself.

Personally I think things will take a step up at WWDC 2025 and we may start to see features introduced that take M1-M3 based Macs to their limits. Fast forward another year to WWDC 2026 and I think we start seeing A.I. features that are exclusive to M4 or above.

 

[zarathu]

So far.... I find that zarathu intelligence beats out apple intelligence hands down.

 

[Gnattu]

Just look at the iPhones. Not very impressive in their TOPS ratings, and yet they outperform pretty much anything in on-device ML tests.

If you are talking about the NPU maybe, but the SME coprocessor has some very AI-optimized design and probably the fastest SME implementation for i8i32 operations. I performed some theoretical throughput test on the A18 Pro SME and found that the i8i32 is ridiculously fast:

A18 Pro SME Test

A18 Pro SME Test. GitHub Gist: instantly share code, notes, and snippets.

gist.github.com

16 TOPS i8i32 outer product on CPU extremely fast and probably nothing like AI workloads that uses 8bit integer matrixes extensively so this is really designed for AI in my opinion.

 

[leman]

If you are talking about the NPU maybe, but the SME coprocessor has some very AI-optimized design and probably the fastest SME implementation for i8i32 operations. I performed some theoretical throughput test on the A18 Pro SME and found that the i8i32 is ridiculously fast:

A18 Pro SME Test

A18 Pro SME Test. GitHub Gist: instantly share code, notes, and snippets.

gist.github.com

16 TOPS i8i32 outer product on CPU extremely fast and probably nothing like AI workloads that uses 8bit integer matrixes extensively so this is really designed for AI in my opinion.

Unfortunately there is an error in my SME benchmarking code that you have presumably used. I calculate the FLOPS of widened outer products incorrectly. The actual figure for i8i32 is only 4TOPS. I’ll upload new code with detailed analysis by the end of the week.

 

[leman]

There is no large jump in TOPS with M4. It’s less than 10% faster than M3, they’re just using different metrics.

It is more nuanced. M3 and earlier has the same rate for FP16 and INT8 operations. M4 has doubled INT8 operations. Different metric, yes, but using that metric would not change the performance of M3.

We don’t see it that much in benchmarks yet because most of them use FP16 weights. Also, the software side was slower in taking the advantage of new capabilities. As the models get optimized, the performance will go up.

BTW, you can see it in new GB6 results here: https://browser.geekbench.com/search?k=ai&q=Mac16 Note how quantized scores running on the NPU are around 50K? M3 family is at ~35K. That’s almost 50% faster.

 

[novagamer]

It is more nuanced. M3 and earlier has the same rate for FP16 and INT8 operations. M4 has doubled INT8 operations. Different metric, yes, but using that metric would not change the performance of M3.

We don’t see it that much in benchmarks yet because most of them use FP16 weights. Also, the software side was slower in taking the advantage of new capabilities. As the models get optimized, the performance will go up.

BTW, you can see it in new GB6 results here: https://browser.geekbench.com/search?k=ai&q=Mac16 Note how quantized scores running on the NPU are around 50K? M3 family is at ~35K. That’s almost 50% faster.

Thanks, I did miss this. INT16 / 32 is a minimal speedup though correct +/- 10%? I hate that you can’t sort those charts!

The INT8 speedup is nice, nvidia did something like this a few years ago if I remember correctly.

I remember back in 2018 I was only concerned with FP64 and bought a Radeon VII due to it dominating there, man was that misguided in hindsight. I think I got ROCm to work correctly maybe twice.

 

[name99]

I think Apple is so far behind on AI, that by the time there's any new and exciting features worth looking forward to, we'll be shopping for M6 and M7 Macs.

"AI" is a huge subject and there is no single "being ahead".
Apple is not competing in the google-like space, ie asking an LLM random questions, just like they didn't compete in the google-like space of asking a webcrawler random questions.

But there are other things you want from an AI.
One is having AI functionality available in all relevant APIs. This is the world of MS, Google (Android), and Apple and no-one really has this settled yet. It's like GUIs in 1980 -- sure, it's gonna happen, but no-one knows what it will look like. But things like Vision Pro probably help Apple in this respect.

Another is personalization. Apple has this today in the sense that Siri/AI has access to your email, calendar, contacts, etc, and (more or less, and presumably improving every few months) can use these to make smarter links and replies. Something like MS Recall makes this much better, but the hysteria around MS Recall seems to have cut off that area of improvement for a year or three. I hope Apple will offer similar functionality, and maybe they will be able to do so in a way that allows the hysterics to just ignore it and the rest of us to make full use of it.

Another dimension of personalization is writing in your style. No-one has this right now, but it's probably easier for Apple to do than the other suspects. They have more samples of your writing than eg ChatGPT or Claude, and it's easier to do the personalization on-device (eventually...) than to run huge per-user models in a data warehouse.

Then you get merges of the two above ideas, for example tying together my writing style and what I tend to read and do (ie Recall) to provide personalized summaries and non-garbage recommendations. What I want from say a summary of an article about Apple (tech details) is different from what a finance person wants from such a summary. And what I want as recommendations (articles, movies, etc) seems to be very different from what most people like; I'm unimpressed with any recommendation system I've used, and it doesn't help that I can't feed into the system "THIS is the reason I disliked this movie, and I don't want to see any other movie with this in it".
Not to mention that pretty much any non-Apple recommendation system will have some way for money to tip the scales (Google routes users to recent movies paying payola, Netflix routes to movies they own, etc). Apple may well corrupt their system in the same way, but at least it has the *potential* to operate purely on the basis of what works best for each user.