Apple Silicon deep learning performance

[GrumpyCoder]

I don't do much speech, mostly vision, but... memory. The A5000 runs circles around the M4 Max for compute, but it's limited to 24GB.

Also, it's ironic that Intel MBPs ran so hot, loud and throttled and now we're in the same situation with the 14" MBP (at least throttling as it seems). There were already complains about noise with the M3 Max in the 14". Looks like M4 Pro is the new best choice for 14", but that also means back to a mobile + a desktop Mac instead of using the maxed out MBP for everything.

 

[Gnattu]

Also, it's ironic that Intel MBPs ran so hot, loud and throttled and now we're in the same situation with the 14" MBP (at least throttling as it seems). There were already complains about noise with the M3 Max in the 14". Looks like M4 Pro is the new best choice for 14", but that also means back to a mobile + a desktop Mac instead of using the maxed out MBP for everything.

Back in the Intel days you don't even have a choice to put the same chip into the 13 inch as the 16 inch, and the 13 inch has less than half of the performance of the 16 inch one, and the fact that we can have such Max chip in 14 inch is already a big win.

The maxed out MBP 16 inch does not throttle with M3 Max (with maxed fan) and I think it will not throttle with M4 Max either as the M4 Max actually improved the thermal a bit. So if you want to sacrifice the portability a bit you can get the 16 inch one.

The core problem here is that the Max GPU is too big for 14 inch to run at full clock sustained and unfortunately is the case for machine learning workloads. I personally still prefer 14inch though, because the 16inch is a bit too heavy to carry every day, at least in my opinion it is a bit too heavy.

I'm still excited to have the choice to run 70B models on a 14inch laptop with usable token generation speed, and MBP 14 is still the only choice at the moment due to the aggressive VRAM limit Nvidia is enforcing on its laptop chips.

 

[leman]

Also, it's ironic that Intel MBPs ran so hot, loud and throttled and now we're in the same situation with the 14" MBP (at least throttling as it seems). There were already complains about noise with the M3 Max in the 14". Looks like M4 Pro is the new best choice for 14", but that also means back to a mobile + a desktop Mac instead of using the maxed out MBP for everything.

Well, Intel chips were already heavily throttling when opening a spreadsheet. The fan would go turbo in single-threaded tasks. Throttling with Intel chips meant that the high-end CPU models were often slower than midrange ones. It’s really not the same thing.

 

[theorist9]

Also, it's ironic that Intel MBPs ran so hot, loud and throttled and now we're in the same situation with the 14" MBP (at least throttling as it seems). There were already complains about noise with the M3 Max in the 14". Looks like M4 Pro is the new best choice for 14", but that also means back to a mobile + a desktop Mac instead of using the maxed out MBP for everything.

Sorry to pile on with everyone else (I like your posts!), but I don't believe this is an analogous situation. I checked the Wikipedia table for the last generation of Intel MBP's (Touch Bar, 2016-2020), and the 13" always got a much lower-TDP class of processor (U or NG7-series) than the 15" (H, HQ, or HK-series).

So it's only because Apple has been able to offer the same class of CPU/GPU in the 14" and 16" AS MBP's that we're starting to see significant thermal issues with the maxxed-out (no pun intended) 14".

A more direct analog to the Intel era would be a 14" AS Pro (or even 14" base AS) vs. a 13" U/NG7-series, and a 16" AS Max vs. a 15" H/HQ/HX-series.

Source:

MacBook Pro (Intel-based) - Wikipedia

en.wikipedia.org

 

[dmccloud]

Back in the Intel days you don't even have a choice to put the same chip into the 13 inch as the 16 inch, and the 13 inch has less than half of the performance of the 16 inch one, and the fact that we can have such Max chip in 14 inch is already a big win.

The maxed out MBP 16 inch does not throttle with M3 Max (with maxed fan) and I think it will not throttle with M4 Max either as the M4 Max actually improved the thermal a bit. So if you want to sacrifice the portability a bit you can get the 16 inch one.

The core problem here is that the Max GPU is too big for 14 inch to run at full clock sustained and unfortunately is the case for machine learning workloads. I personally still prefer 14inch though, because the 16inch is a bit too heavy to carry every day, at least in my opinion it is a bit too heavy.

I'm still excited to have the choice to run 70B models on a 14inch laptop with usable token generation speed, and MBP 14 is still the only choice at the moment due to the aggressive VRAM limit Nvidia is enforcing on its laptop chips.

Apparently Apple made a choice to keep fan speeds down in the 14" models, which can lead to throttling. I have seen a couple of YouTube videos where the reviewers used TGPro to keep the fans at higher RPMs than Apple's fan curves, and the M4 was not throttling as a result. The question for any user would be whether the increase in fan/exhaust noise is worth the performance boost.

 

[flybass]

Apparently Apple made a choice to keep fan speeds down in the 14" models, which can lead to throttling. I have seen a couple of YouTube videos where the reviewers used TGPro to keep the fans at higher RPMs than Apple's fan curves, and the M4 was not throttling as a result. The question for any user would be whether the increase in fan/exhaust noise is worth the performance boost.

Question - why wouldn't apple allow the fans to ramp up with excessive heat? Isn't that the whole point of dynamic fan speeds?

 

[name99]

Back in the Intel days you don't even have a choice to put the same chip into the 13 inch as the 16 inch, and the 13 inch has less than half of the performance of the 16 inch one, and the fact that we can have such Max chip in 14 inch is already a big win.

The maxed out MBP 16 inch does not throttle with M3 Max (with maxed fan) and I think it will not throttle with M4 Max either as the M4 Max actually improved the thermal a bit. So if you want to sacrifice the portability a bit you can get the 16 inch one.

The core problem here is that the Max GPU is too big for 14 inch to run at full clock sustained and unfortunately is the case for machine learning workloads. I personally still prefer 14inch though, because the 16inch is a bit too heavy to carry every day, at least in my opinion it is a bit too heavy.

I'm still excited to have the choice to run 70B models on a 14inch laptop with usable token generation speed, and MBP 14 is still the only choice at the moment due to the aggressive VRAM limit Nvidia is enforcing on its laptop chips.

If this is a real problem (I don't think we know that it is, just people assuming this) the obvious solution is that going forward Apple only sells the 14" with 32 GPU cores (or the equivalent with the M5).

There are TWO M4 Max's, and if the biggest one doesn't fit, just don't use it!

 

[name99]

Question - why wouldn't apple allow the fans to ramp up with excessive heat? Isn't that the whole point of dynamic fan speeds?

About Power Modes on your Mac - Apple Support

Power Modes allow you to tailor your Mac's performance to your workload.

support.apple.com

"

High Power Mode​

High Power Mode allows the fans to run at higher speeds. The additional cooling capacity may allow the system to deliver higher performance in very intensive workloads. When High Power Mode is enabled, you may hear additional fan noise.
"

Of course YouTubers gotta get them clicks of outrage, regardless of whether Apple already provides a solution or not...

 

[flybass]

About Power Modes on your Mac - Apple Support

Power Modes allow you to tailor your Mac's performance to your workload.

support.apple.com

"

High Power Mode​

High Power Mode allows the fans to run at higher speeds. The additional cooling capacity may allow the system to deliver higher performance in very intensive workloads. When High Power Mode is enabled, you may hear additional fan noise.
"

Of course YouTubers gotta get them clicks of outrage, regardless of whether Apple already provides a solution or not...

If I'm reading correctly, high power mode was not available on the 14 inch until M3. So then I'm guessing for M2 or below, fans can already kick up to max or this is an omission to help with marketing new models.

Edit: Read a bit more -- so probably heat dissipation wasn't good enough on earlier 14 inch generations.

 

[Gnattu]

If this is a real problem (I don't think we know that it is, just people assuming this) the obvious solution is that going forward Apple only sells the 14" with 32 GPU cores (or the equivalent with the M5).

There are TWO M4 Max's, and if the biggest one doesn't fit, just don't use it!

That is not the solution because that also reduced memory bandwidth and it also has less CPU cores. You don’t have to be afraid of GPU throttling that much. A big GPU works at lower clock will still have efficiency advantages and Nvidia is doing the same on some of its SKUs, those are just hard locked to a lower frequency. You can think of the 14 inch one as the same. When you choose a form factor you have to accept the trade off. You don’t always push GPU to its max but the form factor is always there. I really like the portability of the 14 inch compared with the 2KG 16 inch

 

[Bungaree.Chubbins]

If I'm reading correctly, high power mode was not available on the 14 inch until M3. So then I'm guessing for M2 or below, fans can already kick up to max or this is an omission to help with marketing new models.

Edit: Read a bit more -- so probably heat dissipation wasn't good enough on earlier 14 inch generations.

On the M3s I think it's exclusive to the M3 Max. I wish it were available on my M3 Pro.