Will Apple silicon benchmarks matter as much going into the future

[the8thark]

As we know the Apple silicon Macs will have high performance cores and power efficient, lower performance cores. Will benchmarking them even be comparable to whatever intel puts out in the future? I've been thinking about this and here's a few points I've come up with.

1. For single core benchmarks, you can benchmark any intem core you want to, they are all the same. You get your result.
For Apple silicon though, a performance core will give a very different result to the low power core. Sure you could test the low power cure just hard enough so the benchmarking process doesn't transfer to the high performance core. That's totally missing the point of the low poer cores though. The benchmark software developers will ahve to add that code in so we can choose which Apple silicon core we are benchmarking.

2. For multicore benchmarking, on Intel it's easy. Just benchmark every core working hard and you have your result.
On Apple Silicon what do we do? Do we just benchmark every core together and just take the average result of both types of cores? Or do we only benchmark the high performance cores and divide that by the number of cores so get a "per core" result when using multicores at the same time?

3. Apple moving to the single SoC. That's happening, however what about the future? In the future what we consider to be a CPU and GPU could be transitioned into one larger whole. Not like the current integrated solution, more like a different, implimentation of that that doesn't have both as two parts of the same whole. It would be just one whole doing both processes. The process making this happen would not be comparable to the standard dedicated CPU and GPU we are used to today. In Intel and co stick with the dedicated model, then comparing this to Apple silicon would be quite impossible indeed.

4. Apple could make improvements to enhance the overall user experience that makes your apps run even better. These improvements could be through optimisation of some kind, not just faster better cores with more flops. (I say flops and not tflops as eventually we will one day hit more than 1000 flops). This could make an Apple silicon Mac with a lower benchmark score the better PC to use.

What are your opinions on this?

 

[ader42]

You make some good points. Apple Silicon is a new paradigm in computing in my opinion.

It will become harder and harder to compare AS with traditional PC cpu/gpu as Apple innovate more and more and make discrete entities for specific tasks, such as Neural Engine and M11.

So traditional benchmarks may become less and less relevant In that respect. As you say we might have Macs that run rings around PCs but score unfavourably on some set tests that don’t reflect real world use.

We might end up with certain common tasks that are greatly enabled by M11 or Neural Engine, and just having these might makes Macs far superior to PCs for certain tasks, i.e. a low end Mac being superior to a high-end PC for specific tasks. Look how good the iPad Pro is for video editing compared to many desktops for example.

 

[Kostask]

You raise interesting points, but eh big picture is that benchmarks only tell you how well benchmarks perform on a platform. In many cases, it isn't a reflection of real world performance.

One of the most common benchmarks is Cinebench. Depending on the platform, you can get some pretty good numbers on it. However, in the real world, those nunbers may be meaningless. This can be seen on the MBP 16". The i9 version can ring up some pretty good numbers, handily beating out the i7 version. In the real world, when editing 4K video for example, the i9 will often thermally throttle, where the i7 won't, or wil throttle less, so the i9 version and the i7 version are a lot closer in the real world than Cinebench would have you believe.

I have an 17-8700K, and a dual Xeon (E5-2680) workstation (both running Windows 10). On every benchmark the i7 machine absolutely destroys the dual Xeon machine. In the real world, it is often that the dual Xeon machine that is faster, and less prone to slowing down.

If your entire workload consists of running benchmarks, then go and find the machine with the highest benchmark performance. If you run things like video editors, don't rely on benchmarks so much.

Eventually, the benchmark writers will figure out what to do about AS SoCs, and rewrite their benchmarks accordingly.

 

[leman]

1. For single core benchmarks, you can benchmark any intem core you want to, they are all the same. You get your result. For Apple silicon though, a performance core will give a very different result to the low power core. Sure you could test the low power cure just hard enough so the benchmarking process doesn't transfer to the high performance core. That's totally missing the point of the low poer cores though. The benchmark software developers will ahve to add that code in so we can choose which Apple silicon core we are benchmarking.

Depends on what you want, doesn't it? Usually, when one talks about single core performance one means peak performance. So only high performance cores are relevant here.

2. For multicore benchmarking, on Intel it's easy. Just benchmark every core working hard and you have your result.
On Apple Silicon what do we do? Do we just benchmark every core together and just take the average result of both types of cores? Or do we only benchmark the high performance cores and divide that by the number of cores so get a "per core" result when using multicores at the same time?

You load all the cores and measure the time it takes to complete the task. Again, with these things one usually cares about computing the task as soon as possible. If you want to measure something different, say, the efficiency of the low-power cores, you will need a different benchmark paradigm of course.

3. Apple moving to the single SoC. That's happening, however what about the future? In the future what we consider to be a CPU and GPU could be transitioned into one larger whole.

Why does it matter? As long as it follows the usual software model we can just keep running the usual benchmarks.

 

[leman]

You make some good points. Apple Silicon is a new paradigm in computing in my opinion.

What is new about it? SoC approach has been there for years, everyone uses it. There were much more radical attempts, like Intel's Larrabee (the kind of CPU+GPU fusion @the8thark speaks about) or the Cell CPU, but they didn't come too far. Lackluster performance, too difficult to program efficiently, not power efficient.

Apple's approach is a fairly traditional CPU+GPU, they just pack a bunch of auxiliary coprocessors and a good unified memory subsystem. What is "new" about their approach is that they also control the rest of the platform (including computer design and the OS), which allows them to put everything together into a really tight and efficient package.

 

[q64ceo]

Benchmarks dont really matter. They're synthetic.

 

[burgerrecords]

Even with the challenges, benchmarks are going to be more significant than before for the change. When mac was selling cpu gpu ram etc that you could essentially get elsewhere, you knew just about exactly what you were getting.

If I recall correctly, Mac used to advertise benchmarks in the PowerPC era?

I don’t expect Apple computers are going to be so radically different as to make comparison irrelevant.

As to efficiency cores, if those reduce processing time of a workload then great. You can benchmark a n core processor with and without multithreading and you can measure that difference even though there is no exact analog on Apple silicon l don’t believe. You can measure 8 core vs 16 core etc. As with PowerPC some tasks at various lifecycles will probably be faster on Mac and some will be probably be faster on AMD and some on Intel etc.

 

[Joelist]

The real question is not going to be answered with synthetic benchmarks. You'll have to identify real world tasks and do them on the different systems. It's the only way to account for the entirety of the Apple package.

 

[Kpjoslee]

I have yet to see processors that does well on Geekbench or Cinebench happens to be poor performer in real world applications. (also depends on various other factors but considering everything else is equal). While it doesn't tell everything about how some of the real world apps will perform(due to their own limitations), I would say benchmarks are pretty good indicator of where the one particular configuration stand compared to the others.

 

[leman]

Benchmarks dont really matter. They're synthetic.

The real question is not going to be answered with synthetic benchmarks. You'll have to identify real world tasks and do them on the different systems. It's the only way to account for the entirety of the Apple package.

Synthetic benchmarks are a good way to evaluate aspects of various computing subsystems. And other, more complex benchmarks can be a decent proxy for certain class of tasks*. Of course m, if you are buying a computer for a specific purpose, you should always evaluate its performance for Workflows that are relevant fir you. But that is a poor proxy of the overall compute capability as well...

Except Geekbench. Screw geekbench.

 

[the8thark]

Again, with these things one usually cares about computing the task as soon as possible.

This to me is the whole point of what you said and I agree with you 100%. In my opinion, benchmarking this kind of thing (ie what the users actually care about) would require app specific benchmarks. Getting a higher number in geekbench or similar might not always mean your computing task is completed quicker and more efficiently.

My original post here was about the overall benched numbers (ie geekbench or similar). However in app specific benchmarks will always be relevant as they take into account the entire mac, not just a number of the main cores.

Synthetic benchmarks are a good way to evaluate aspects of various computing subsystems. And other, more complex benchmarks can be a decent proxy for certain class of tasks*. Of course m, if you are buying a computer for a specific purpose, you should always evaluate its performance for Workflows that are relevant fir you. But that is a poor proxy of the overall compute capability as well...

Except Geekbench. Screw geekbench.

Agreed, the synthetic benchmarks are still nice to have, but only when pared with the app/workflow specific benchmark that relates to the work you do. I feel having both ni the future will be needed. They exist today but many people just look at the overall number and call it a day. People who do this are missing out on a lot of very useful information that might matter to them.

Agreed, geekbench . . . well I agree with you there.

 

[DearthnVader]

In a lot of ways, fully synthetic benchmarks are the lies we agree to tell each other.

Be that as it may, we all want to get some idea as to the speed of our computers in comparison to a computer we are replacing it with, or a computer we could have replaced it with. I think we all want to know if Apple SoC is going to be truly faster than the Intel system it replaces.

Benchmarks give us a quick and dirty idea, but that may not always be an accurate idea, tho sometimes perception is reality.

I often think, because hardware moves so fast, that software often runs about 10 years behind. When the Apple SoC Macs ship, everyone and their brother will be doing Apple's to Apples comparisons to Intel systems, but software has been optimized for Intel specific instructions for 15 years, and by the time most software has been optimized for Apple SoC, no one will really care about how things could have run on Intel.

Apple controls what it can control, the core OS and the apps that ship with the core OS, it's up to the third parties to optimize for Apple SoC, so I think we have to find benchmarks that are not synthetic, that, as much as possible simulate real world workflows.

 

[leman]

I often think, because hardware moves so fast, that software often runs about 10 years behind. When the Apple SoC Macs ship, everyone and their brother will be doing Apple's to Apples comparisons to Intel systems, but software has been optimized for Intel specific instructions for 15 years, and by the time most software has been optimized for Apple SoC, no one will really care about how things could have run on Intel.

Very few programs go through the trouble of optimizing the code for a specific CPU. This is usually done by the compiler. And even then, Intel≠Intel. Different families of Intel CPUs need different kinds of optimizations.

In the end, I’d argue that most software is fundamentally very poorly optimized for any CPU. The dominant programming techniques of last decades tend to produce terrible memory access patterns.

 

[TylerL]

I think the conversation about "benchmarks don't matter" will get a lot louder as Windows/x86 pundits go through their stages of grief over Apple Silicon performance outclassing anything Intel and AMD can provide.

I predict a level of despondence among those in the traditional computing world that has not been seen since the months after the iPad launch.

 

[DearthnVader]

I think the conversation about "benchmarks don't matter" will get a lot louder as Windows/x86 pundits go through their stages of grief over Apple Silicon performance outclassing anything Intel and AMD can provide.

I predict a level of despondence among those in the traditional computing world that has not been seen since the months after the iPad launch.

I wish I shared your confidence, I'm more than a little skeptical that Apple SoC's can out class performance levels of x86.

That having been said, I understand why Apple is making this move, and I don't have any code that won't run or compile on Arm, so I'm hoping Apple can cut price a little on the type of portable I'm in the market for.

 

[Joelist]

What Apple is doing is different than anyone else right now. The control the entire hardware package AND the software package at the OS level. This is one reason iOS apps are fast and stable - Apple controls exactly how they can access system resources. And when they go to AS for the Mac expect them to be exerting the same type of control at that level.

You could even see hints of it in the WWDC sessions where they told developers that some of their usual tricks of the trade are not allowed anymore under AS and over time more will be in that club - especially anything trying to add to or change kernel or in fact even address the kernel in some spots.

 

[burgerrecords]

What Apple is doing is different than anyone else right now. The control the entire hardware package AND the software package at the OS level

What will be interesting is if this situation progresses from Apple products being “nice to have” to “have to have” (due to actual tech - not arbitrary channel lock in)

I think it will remain similar to how it is now with the iPhone vs android. Just a bit better overall on Apple in almost all ways intangible and tangible, at some costs of $$$ flexibility and openness (but worth it for most of us here) a few programs that work on one platform vs another.

I don’t see materially any difference in stability between platforms. Particularly iOS vs Android are both fine in my experience. Windows isnt very good for people who use it unmanaged or don’t have the knowledge to use it in the right manner - but that’s obviously not a hardware limitation.

(If you are in a developing economy though windows is a lot more user friendly than Linux at little to no cost but both will give you a secure, supported platform to use old hardware while still automatically being patched and the low cost probably outweighs the limitations)

Obviously you can run non Apple systems securely, even if Apple has some nice to have options there too.

Will Apple develop systems in ways to create use scenarios that go beyond “nice to have” because of things you couldn’t otherwise do, due the tight integration of the hardware and software? Or will it just be a bit “better” overall? (cost, flexibility and openness aside)

 

[LeeW]

Synthetic benchmarks don't matter now, they just make people feel warm and fuzzy inside at something appearing to be better than something else whilst ignoring the most important elements. Real world usage.

 

[Kostask]

What will be interesting is if this situation progresses from Apple products being “nice to have” to “have to have” (due to actual tech - not arbitrary channel lock in)

I think it will remain similar to how it is now with the iPhone vs android. Just a bit better overall on Apple in almost all ways intangible and tangible, at some costs of $$$ flexibility and openness (but worth it for most of us here) a few programs that work on one platform vs another.

I don’t see materially any difference in stability between platforms. Particularly iOS vs Android are both fine in my experience. Windows isnt very good for people who use it unmanaged or don’t have the knowledge to use it in the right manner - but that’s obviously not a hardware limitation.

(If you are in a developing economy though windows is a lot more user friendly than Linux at little to no cost but both will give you a secure, supported platform to use old hardware while still automatically being patched and the low cost probably outweighs the limitations)

Obviously you can run non Apple systems securely, even if Apple has some nice to have options there too.

Will Apple develop systems in ways to create use scenarios that go beyond “nice to have” because of things you couldn’t otherwise do, due the tight integration of the hardware and software? Or will it just be a bit “better” overall? (cost, flexibility and openness aside)

Windows, and security have nothing to do with each other. If they did, patch Tuesdays would not be patchiyg over 110 security issues (for 5 months running, anybody want to bet that it isn't going to be six in a row?). Stability is questionable, too, considering that in 3 of those patch Tuesdays, people had either their data irretrievably deleted, or their entire user accounts deleted.

I don't know that Macs will ever be "have to haves" like the iPhone. But I do think that Windows is beginning to show cracks, and very, very big cracks.

 

[burgerrecords]

Windows, and security have nothing to do with each other. If they did, patch Tuesdays would not be patchiyg over 110 security issues (for 5 months running, anybody want to bet that it isn't going to be six in a row?). Stability is questionable, too, considering that in 3 of those patch Tuesdays, people had either their data irretrievably deleted, or their entire user accounts deleted.

It just isn’t that bad if you know what you’re doing or have a corporate managed system. You may have different experiences; my data is anecdotal; but a lot of sensitive data is out there residing on windows servers and clients. Microsoft wouldn’t be the size they were if Windows were doomed.

 

[Kostask]

I do the managing of computers at my company, and it is that bad.

Microsoft is coasting to the market share that Windows has. Have been for many years. They have dismissed their entire internal testing group, over 5,000 people, and replaced them with the Windows Insider folks (I am one of them). As Leo Laporte said in MacBreak Weekly, "Microsoft can't code their way out of a paper bag", direct quote, and it is available on Youtube if you look for MacBreak Weekly. He has also stated on a repeated basis "Friends don't let friends use Windows".

Microsoft is no longer actively managing Windows issues. They have started focusing on stealing people's data (to follow what Google and Facebook already do so successfully), and trying to turn everything into a service (see Office 365), and cloud based services.

The fiinal point is that Microsoft has taken to doing substantial updates to Windows every six months or so. That is why the releases have been ramping up. 1903, 1909, 2004, etc. The thing is, they don't even let the previous version settle down before they are pushing out a new version. The code base for Windows is slowly becoming ever so unstable. And there is no end in sight.

Microsoft became the size it is because of the market share of WIndows and Office in the past. Apple got to the point it is because of the iPhone. Microsoft's market share has next to nothing to do with the quality of its software, and much to do with how cheap the Intel/Windows platform is to buy into. if there were another consumer level OS available on the Intel/Windows platform, maybe Microsoft wouldn't be in the position it is in now. However, in the end, it may be of interest to see where, in terms of market cap, Apple was in comparison to Microsoft, and where they are now.

 

[dogslobber]

Benchmarks are for techies who want to compare Apple and oranges. Most normal users want something that works for their needs and will roll their eyes when you get all techie numbers with them.

 

[tranceking26]

Benchmarks are one thing, what about waiting until they actually release ARM Macs and see what users/reviewers have to say?

 

[burgerrecords]

However, in the end, it may be of interest to see where, in terms of market cap, Apple was in comparison to Microsoft, and where they are now.

In the last 5 years Microsoft stock is up 393% and Apple is up 323%. Obviously you could pick different time periods and argue Microsoft should have been able to accomplish more - but the recent past is not irrelevant.
What is likely more relevant regarding future prospects, is that Microsoft has a higher P/E ratio than Apple while also having a higher dividends yield. Presumably meaning the market expect greater earnings growth from Microsoft than even Apple (in which Windows just coasting from cash cow into oblivion seems unlikely - something needs to drive bing and o365 use)

The last short term there have a been a few issues with updates, market is clearly looking well beyond that and believe whatever Microsoft is doing will provide earnings for many decades to come

I think they both probably are fine.

 

[Tech198]

You make some good points. Apple Silicon is a new paradigm in computing in my opinion.

It will become harder and harder to compare AS with traditional PC cpu/gpu as Apple innovate more and more and make discrete entities for specific tasks, such as Neural Engine and M11.

So traditional benchmarks may become less and less relevant In that respect. As you say we might have Macs that run rings around PCs but score unfavourably on some set tests that don’t reflect real world use.

We might end up with certain common tasks that are greatly enabled by M11 or Neural Engine, and just having these might makes Macs far superior to PCs for certain tasks, i.e. a low end Mac being superior to a high-end PC for specific tasks. Look how good the iPad Pro is for video editing compared to many desktops for example.

Since Apple can do it their way, not the most 'common way' (intel does), who really knows what we could be in for