Geek Benching new iOS updates: WHY?

[roncron]

TL;DR After months of running Geek Bench tests after each iOS update or new beta version, I’m now wondering what we’re supposed to learn from GB.

—

“Professionals” like EverythingApplePro and iAppleBytes run GB tests every time they review a new iOS version/update/beta and use the results as a metric of how fast the iOS version is.

Out of curiosity, I wanted to see if my iPhone X gets similar GB scores.

The attached screenshots show the history, since I bought my iPhone X last November (running 11.2 at that time). I usually run GB after installing a new iOS update or beta version.

This history reveals a few lessons:

1) Regardless of iOS version, single core results usually vary between 4100 and 4300 and multi core usually between 10,000 and 10,400.

2) There’s a lot of variation in the scores, not only from one iOS version to the next but also from different GB runs on the same day using the same device and same iOS version. For example, I ran GB twice on May 29, getting multi core scores of 9848 and 10297.

I imagine that GB scores will depend on what else your phone is doing while GB is running, how long since you last rebooted the phone, etc.

This suggests that you can’t conclude much if the GB score under a new iOS version is different than under the previous version since many factors other than iOS version affect the results, and it seems impossible to control for all of these other factors.

3) The headline feature of iOS 12 is supposed to be speed and stability. Many iOS 12 beta testers—probably over 90%—feel their devices are running faster and smoother under iOS 12 than under iOS 11–both on newer devices like iPX and iPad Pro 10.5, and older devices like iP6 and iPad Air 2. But compare GB scores between the various iOS 12 betas on the one hand and the various iOS 11 versions on the other: They are not systematically different.

For example, my multi core score on the iOS 12 betas has gone up steadily with each beta, from about 10,000 on public beta 1 to nearly 10,500 on public beta 7. But across the various versions of iOS 11, the same benchmark produced scores varying from just under 10,000 to nearly 10,600.

My single score scores tell a similar story: They are almost always between 4100 and 4300 regardless of which version of iOS 11 or iOS 12 I’m running.

I know way less about this stuff than most of you. Am I correct in concluding that GB results cannot really tell us much about how much better a new version of iOS is than the previous version? Isn’t it true that GB results depend far more on your hardware and whatever processes may be running in the background than on the particular iOS version?

And if this is correct, why do EverythingApplePro and similar reviewers waste our time and theirs with GB results?

[doublepost=1534960629][/doublepost]To me, it seems like the main value of GB is to see how much faster a new device is compared to the previous model.

And even then, I wouldn’t feel comfortable comparing a single GB run on each device. Rather, I’d want to see the average GB scores from 5 or 10 runs on each device.

I applaud the GB programmers for showing us the average score for each different device type from the latest iPhone X all the way back to iPhone 4S.

 

[GreyOS]

I do not trust GB to really say much about iOS performance.

As an operating system, iOS handles a lot of the low level stuff, interfacing with the CPU, RAM, GPU, storage etc., so improvements at that level can have an impact on all general operations on the system, including those performance by GB. See how the fixes for Meltdown lowered performance across a range of operating systems because they had the change the way it interacted with the CPU at a low level.

But on the other hand, a lot of optimisations are probably not that low down. For example, I doubt the improvements they made to keyboard loading speed were low level. They probably just found that the keyboard software they wrote was inefficient in places. This isn’t something that GB would test for because it doesn’t invoke the keyboard (just as an example)

It is certainly possible to imagine all of the optimisations Apple did in iOS 12 were in isolated high level routines the operating system runs and not something GB could really test.
[doublepost=1534962152][/doublepost]Also the fact you put “Professionals” in quotes answers your own question I think

 

[LoveToMacRumors]

TL;DR After months of running Geek Bench tests after each iOS update or new beta version, I’m now wondering what we’re supposed to learn from GB.

—

“Professionals” like EverythingApplePro and iAppleBytes run GB tests every time they review a new iOS version/update/beta and use the results as a metric of how fast the iOS version is.

Out of curiosity, I wanted to see if my iPhone X gets similar GB scores.

The attached screenshots show the history, since I bought my iPhone X last November (running 11.2 at that time). I usually run GB after installing a new iOS update or beta version.

This history reveals a few lessons:

1) Regardless of iOS version, single core results usually vary between 4100 and 4300 and multi core usually between 10,000 and 10,400.

2) There’s a lot of variation in the scores, not only from one iOS version to the next but also from different GB runs on the same day using the same device and same iOS version. For example, I ran GB twice on May 29, getting multi core scores of 9848 and 10297.

I imagine that GB scores will depend on what else your phone is doing while GB is running, how long since you last rebooted the phone, etc.

This suggests that you can’t conclude much if the GB score under a new iOS version is different than under the previous version since many factors other than iOS version affect the results, and it seems impossible to control for all of these other factors.

3) The headline feature of iOS 12 is supposed to be speed and stability. Many iOS 12 beta testers—probably over 90%—feel their devices are running faster and smoother under iOS 12 than under iOS 11–both on newer devices like iPX and iPad Pro 10.5, and older devices like iP6 and iPad Air 2. But compare GB scores between the various iOS 12 betas on the one hand and the various iOS 11 versions on the other: They are not systematically different.

For example, my multi core score on the iOS 12 betas has gone up steadily with each beta, from about 10,000 on public beta 1 to nearly 10,500 on public beta 7. But across the various versions of iOS 11, the same benchmark produced scores varying from just under 10,000 to nearly 10,600.

My single score scores tell a similar story: They are almost always between 4100 and 4300 regardless of which version of iOS 11 or iOS 12 I’m running.

I know way less about this stuff than most of you. Am I correct in concluding that GB results cannot really tell us much about how much better a new version of iOS is than the previous version? Isn’t it true that GB results depend far more on your hardware and whatever processes may be running in the background than on the particular iOS version?

And if this is correct, why do EverythingApplePro and similar reviewers waste our time and theirs with GB results?

View attachment 777268
View attachment 777269
[doublepost=1534960629][/doublepost]To me, it seems like the main value of GB is to see how much faster a new device is compared to the previous model.

And even then, I wouldn’t feel comfortable comparing a single GB run on each device. Rather, I’d want to see the average GB scores from 5 or 10 runs on each device.

I applaud the GB programmers for showing us the average score for each different device type from the latest iPhone X all the way back to iPhone 4S.

Longer videos, more money, more ads, shyt like that

 

[JohnnyW2K1]

It’s pretty much absolute nonsense. Geekbench, like any CPU benchmarking, is more of a measure of the hardware (the actual CPU chip inside your phone) than of the OS.

It’s so silly that GeekBench has become so ubiquitous in iOS reviews. The only way you’d a huge swing is if there was a bug in iOS (hindering the running of software), otherwise the differences are not really noteworthy.

GeekBench is good for comparing CPUs (eg. A9 vs A10) but to use it to compare iOS versions, or even sillier, iOS betas, is a waste of time.

 

[Michael Goff]

Videos use it because the viewers want it. But Geekbench has no value period.

 

[Yun0]

Alot of EverythingApplePro's videos (and may other Apple channels) are crap and baity for their youtube money, geekbench "drops" (more like error margin) by even less than 8, "this beta is bad its stuttering more its more laggy, IDK what Apple is doing, my score is worse" bla bla, then all the sheep instant believe and obey cause someone with millions of subscribers said something, so it must be accurate. Another point EAP loves to pointlessly throw out there every single damn time is how 90% of updates "always free up storage", TLDR, the guy knows nothing about caching, temp files etc..

 

[perezr10]

I think running a benchmark like Geekbench is kind of like taking your car to the drag strip and running the quarter mile. Your engine horsepower may not ever change but your ET (elapsed time) will, depending on conditions and weight.

When Apple optimizes iOS it is analogous to reducing weight. Your “car” runs the 1/4 quicker due to a lighter weight. An increased benchmark score is a measure of how much more optimized the iOS is. Depending on conditions, just like racing.

FWIW, I like EAP. He just seems like a likable nice guy who’s willing to break his own iPhones for our viewing pleasure.

 

[GreyOS]

I think running a benchmark like Geekbench is kind of like taking your car to the drag strip and running the quarter mile. Your engine horsepower may not ever change but your ET (elapsed time) will, depending on conditions and weight.

When Apple optimizes iOS it is analogous to reducing weight. Your “car” runs the 1/4 quicker due to a lighter weight. An increased benchmark score is a measure of how much more optimized the iOS is. Depending on conditions, just like racing.

Nah I think most optimisations are more like getting a better driver. The car is the same but the driver uses it more intelligently. When Geekbench test they put their own driver in the car.

 

[perezr10]

Nah I think most optimisations are more like getting a better driver. The car is the same but the driver uses it more intelligently. When Geekbench test they put their own driver in the car.

Ha! Ive seen many times where swapping drivers could produce better times due to greater experience. I can see that argument too.

However, I see the processor as “pushing” code to get things done. Code that isn’t optimized is “heavier” for the processor to push.