True. I just wanted to compare prices. V100 16GB still retailed for over $10,000. That's pretty insane. You can almost get 2x Mac Studios with M1 Ultra and 128GB of video ram.
A100 is $14000.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Apple Silicon deep learning performance
- Thread starter gl3lan
- Start date
- Sort by reaction score
A100 is $14000.
WRT to ML on workstations, laptops and 'edge' devices (phones/glasses):
Wouldn't the most important devices (to Apple) be those edge devices where a goal might be to generate models (the learning side) on the fly as opposed to just applying them? If so, wouldn't that mean the Apple would always lean towards more efficiency?
Why would Apple prioritize ML on a M based laptop that would overheat, drain the battery and not compete with Nvidia's 250W+ offerings?
All Apple has to do now is figure out a way to need to do ML on those edge devices.
Wouldn't the most important devices (to Apple) be those edge devices where a goal might be to generate models (the learning side) on the fly as opposed to just applying them? If so, wouldn't that mean the Apple would always lean towards more efficiency?
Why would Apple prioritize ML on a M based laptop that would overheat, drain the battery and not compete with Nvidia's 250W+ offerings?
All Apple has to do now is figure out a way to need to do ML on those edge devices.
Apple does this through transfer learning. It trains general-purpose models on large computers and partially retrains them on edge devices based on user input. For instance, virtual assistants adapt to your vocabulary and accent when you speak to them.
Hey Luca, nice work! However, when launching the benchmarking script on my M1 Max I am running into the issue described here (which I was indeed able to replicate): https://github.com/pytorch/pytorch/issues/78001.
Using exactly the same setup as in the repo. Did anyone else run into this?
Doesn't that support the notion that from Apple's marketing perspective, there's no priority to try and compete with the processing power of those workstations?
Transfer learning scenarios would warrant ML on the battery-powered edge... and in that black box of Apple's data centers.
Huh, how is that not bad? It’s painfully slow on larger nets and datasets. And no, no one cares about power consumption. This is about getting things up and running locally as fast as possible and then submit it to larger systems doing the heavy lifting and those eat up 500kW+ anyway. One day late and you might lose millions of $ in funding. So no, power consumption doesn’t matter in this case. Apple needs to do better with future systems, especially the Mac Pro replacement.
I do wonder, though, how relevant that will ultimately be for a professional using a laptop (in the case of a desktop workstation/MP, then yeah I see the point and the use case). I have been working in ML for a fairly long time, and this stuff has become so demanding as of late that the amount of computation we do locally on our laptops has actually decreased year after year, to the point that the raw power of the machine on the desk is irrelevant 95% of the time.
It's great that native GPU support is now available as we do value the ability to prototype and debug a small-scale model locally before offloading it to a remote system. That's useful. But anything more than that is sent to the cloud anyway. There is little reason to play around with really large nets, and local speed doesn't really matter, if a job is time-critical adequate machines are used for that.
We usually get things up and running locally to the point we know it's going to work and then go to either a cloud system or local cluster. The local cluster is usually very busy, that's why we can't waste time on it settings things up. Cloud is nice, but we can't always go there. It depends on the data. We have to follow very strict rules in some cases.
As for laptops, I'm running a Razer with RTX5000 which is usually enough for this. I can play a little more with my desktop with RTX8000 and even run training for smaller datasets/nets there. Rest goes to larger systems (Nvidia or Dell) then.
Yeah I know not everything can go on the cloud because of data regulations, I have worked on projects like that. Still, even in those cases we ran jobs on an internal server or workstation equipped with multiple GPUs, never on laptops.
Then again, all my company-issued computers were Intel MBPs so local training on GPU wasn't really possible anyway lol.
Let's hope the gap shrinks a lot. ML models are getting bigger and bigger by the day and training times are shooting up and up. Even on my deskside with 2 NVidia 3070s it can take hours to train some of my models. I could send them to a cloud GPU or TPU, but because of costs, I usually only do that when I am fine tuning, not experimenting.
Getting Pytorch to play nicely will pay dividends for not only M1 Max but for M1 Ultra, 2x M1 Ultra (Mac Pro), M2, M3, M4 and so on. It'll help Apple sell more Mac Studios and Pros. Basic testing can be done on M laptops.
Apple Silicon could potentially become an excellent valued option for ML because of its unified memory architecture. At some point in the next few years, you might be able to get 512GB of unified memory via the Mac Pro. You can't get this much vRAM today unless you're renting super expensive clusters or spend millions on local hardware.
And who knows? In the future, Apple could release Apple Silicon Cloud:
You're Apple. You just built a 40-core SoC for Mac Pro. Now what?
Mark Gurman is saying that Apple is working on a 40-core SoC for the Mac Pro for 2022. You're Tim Cook, sitting in his nice office, looking at how much money you just spent to make this giant SoC for a relatively small market. In fact, you have to do this every year or every two years to keep...
forums.macrumors.com
Last edited:
Not sure Apple will ever want to go the Cloud route but I agree that they need to up the ante for the Mac Pro platform and bring some feature parity on the GPU side, lest they want it to be just a "brand statement" computer confined to a niche of professional video producers. The potential is there with the AS architecture: lots of unified memory that can be accessed by the GPU, high bandwidth, low latency. But it needs software support.Getting Pytorch to play nicely will pay dividends for not only M1 Max but for M1 Ultra, 2x M1 Ultra (Mac Pro), M2, M3, M4 and so on. It'll help Apple sell more Mac Studios and Pros. Basic testing can be done on M laptops.
Apple Silicon could potentially become an excellent valued option for ML because of its unified memory architecture. At some point in the next few years, you might be able to get 512GB of unified memory via the Mac Pro. You can't get this much vRAM today unless you're renting super expensive clusters or spend millions on local hardware.
And who knows? In the future, Apple could release Apple Silicon Cloud:
You're Apple. You just built a 40-core SoC for Mac Pro. Now what?
Mark Gurman is saying that Apple is working on a 40-core SoC for the Mac Pro for 2022. You're Tim Cook, sitting in his nice office, looking at how much money you just spent to make this giant SoC for a relatively small market. In fact, you have to do this every year or every two years to keep...
I have always been disappointed at how the quarrel with Nvidia resulted in Apple just letting go of ML/AI computing without even trying anymore. It is even more perplexing when you consider that a majority of the scientists and engineers in the field use a Mac as a work machine... I really hope AS can be the trigger for things to turn around.
I agree with you on the second paragraph. Apple gave up a lot by using AMD GPUs lover the last 8 years. Now Apple is just trying to catch up.
Also agreed that video editors aren't going to drive much volume for high-end GPUs. Apple needs ML and AAA gaming in my opinion.
Disagreed on the cloud. I think it's inevitable that Apple will have to offer an Apple Silicon Cloud. The industry is moving towards that direction. There will be a breaking point where businesses will mostly rent high-end workstations from the cloud instead of spending tens of thousands for each machine. This is likely when internet speeds gets faster. For example, I can get a 10G/10G symmetrical fiber to home internet connection for $40 where I live.
You might just be a little biased, but either way everything cloud won’t happen in any currently living person’s lifetime. It’s a little more complicated than throughput equals functionality. The devil is in the details, and there are a lot of devils out there.
Ha! We went from lamp radio to nuclear weapons to CRT to global internet in one persons lifetime. Deep learning was not more than an academic quriosity just a decade ago. And the iPhone has been released in 2007 - folks born that year haven’t even finished school yet.
Don’t underestimate how quickly these things are moving. There is a non-negligible chance that AI will take over and enslave us all before I die
I don't think I'm a little biased.
I'm also referring to workstations, IE, Mac Pros.
If you have a 10g/10g connection, which will very likely happen within your lifetime because it's already here for me, then moving files between your local computer and the cloud computer is fast enough that you might not notice the difference. Or you store the files in the cloud anyways. For businesses, 10g/10g should reach them faster than home internet.
For Macbook Air/Pro, it doesn't make sense to replace them with a Cloud Computer... because you need a computer to access the cloud.
You are probably right that at some point it will be an inevitable move for that segment. One-off buying workstation or server hardware is getting harder to justify by the day as one simply cannot keep up with the speed at which models are becoming greedier in terms of data size and computational power.
But there's a long way to go before Apple can become a viable alternative in the high-end GPU space, they will have to invest in this effort seriously for several years. I hope they don't just decide it is not profitable enough for them to do it and drop the efforts entirely. But I like to remain optimistic for now.
One century of progress is very long, mate. When grandma was born her dad was riding a donkey, 40 years later we landed on the moon, another 30 and we had Internet installed at home. Science and technology will do their job fast, I'm more concerned about the societal and environmental challenges we will have to face to get there
.Apple first needs to improve its support for Tensorflow and Pytorch. Is there any information on what operations/types of neural networks Apple's GPU supports in Tensorflow and Pytorch?
What would be Apple's advantages over current options? Faster than Nvidia GPUs? More cost effective than Habana CPUs? Better software than Google or DataRobot?
Advantages? If all you are about is ML training speed or cost, not much (actually, none whatsoever). From a broader perspective: it's easier to justify getting a superior development platform (a Mac) if it's not that much slower for ML stuff than a comparable crappy Nvidia-based laptop, and it might actually outperform Nvidia if your task is very very large and won't fit in dGPUs memory. That's about it.
Apple has no chance in computing with Nvidia's tensor cores if they use the GPU. As I wrote before, this is comparing general-purpose FP32 units (as efficient as they may be) vs. specialised ML accelerator hardware. The only way how Apple can significantly boost performance here is by expanding their own matrix coprocessors and moving the ML computation there. But then again it doesn't seem like this is where we are going. The ML backends seem to be targeting Metal and not AMX, which suggests that Apple indeed sees the GPU as the primary ML accelerator. What does this mean for the future of AMX? Honestly, no idea. Maybe they plan to more closely integrate the AMX cores and the GPU and expose them via Metal in the future. Maybe the current situation is all we get. Who knows.
The big question is whether Apple sees this stuff as a priority and as part of the AS development roadmap.
I hope so, because if that's not the case the efforts to add support for ML computation on AS GPUs might end up like the ill-fated attempts to do so for AMD: too fragmented, too buggy/limited, too much inertia in favor of existing solutions. In the end everyone gives up on maintaining the libraries and we are back to the Nvidia monopoly.
The community can help, but in the end to keep these projects alive you need a giant behind them.
I hope so, because if that's not the case the efforts to add support for ML computation on AS GPUs might end up like the ill-fated attempts to do so for AMD: too fragmented, too buggy/limited, too much inertia in favor of existing solutions. In the end everyone gives up on maintaining the libraries and we are back to the Nvidia monopoly.
The community can help, but in the end to keep these projects alive you need a giant behind them.
They must, as they dedicate significant resources to this. What worries me more is whether they have a strategy that spans more than one department. It is obvious that their hardware department has low-power inference as a priority, but it’s not clear whether they also work on accelerated training - Apple GPU is not equipped for this and the AMX is simply not enough. I mean, if it’s just the software folks that push these efforts, without contribution from the hardware team, then ML workloads on Apple Silicon will remain a niche use case at best.
That is possible, but fantastical if you consider the path that would be necessary. Similarly, some sort of cloud utopia is so improbable because of the probability of cloud dystopia. Chaos does not favor the cloud. Also, to all of the forum snipers, I’m quite aware of generational advancements. Induction doesn’t work the way you think it does.There is a non-negligible chance that AI will take over and enslave us all before I die
Let's hope Apple focus on ML and AAA gaming. These are literally the biggest reasons for buying and upgrading big GPUs. Continuing to only push video editing for GPU use is meh. Video editing is not at the cutting edge of GPU consumption. Heck, you can do decent Youtuber-style editing on an iPad Pro.
It seems that Apple can now release its Tensorflow plugin the same day Google releases Tensorflow.
Do packages using Tensorflow work out of the box with Apple's Tensorflow plugin?