Phones Samsung tweaking benchmark scores with an overclock when running benchmark apps?

[ChazUK]

Original Source: Definitely worth a full read!

Here's the original source:

http://www.anandtech.com/show/7187/looking-at-cpugpu-benchmark-optimizations-galaxy-s-4

----

Synthetic benchmarks are never a good indication of real world performance on any handset IMO but it appears Samsung is being shady when it comes to Android benchmark applications.

http://www.slashgear.com/samsung-ga...tuned-for-benchmarks-research-spots-30292185/

Samsung tailored its Galaxy S 4 to deliver the best possible scores on popular Android benchmarking tools, investigations have revealed, despite apps potentially not getting the same power for real-world use. The AnandTech research was sparked by claims Samsung was reserving its fastest graphics chip speeds for select benchmarking apps alone, with games and other software only ever seeing slower performance from the Exynos 5 Octa processor found in select models of the Galaxy S 4. The motivation behind the tinkering appears to be to ensure the flagship smartphone posts consistently high benchmarking numbers for comparison with other devices, even if that doesn’t necessarily translate to its everyday abilities.

A very sneaky move from them IMO.

Benchmarks are okay for some but the way I've seen them gamed over the years by ATI and Nvidia over the years on PC (Quack3 Anyone?) makes me think of them as a more secondary metric over other aspects (design, features, price).

Peoples' thoughts on this? Are Samsung being underhanded or is there a legit reason for higher clockspeeds when running a benchmark app?

 

[BaldiMac]

Here's the original source:
http://www.anandtech.com/show/7187/looking-at-cpugpu-benchmark-optimizations-galaxy-s-4

 

[ChazUK]

Here's the original source:
http://www.anandtech.com/show/7187/looking-at-cpugpu-benchmark-optimizations-galaxy-s-4

Just finished reading it. Anand do put together a great read. I'll add the link to the OP, they deserve it.

 

[Assault]

Wow. Like this really matters? Did not seem to matter to anyone before when Apple "lied" about the iPhone 5's CPU being twice as fast as the A5X in the 4S? Actual testing revealed, the iPhone 5 was neither twice as fast, nor faster than the 4S at all in many CPU/GPU aspects. In fact, the 5 was slower in things like speech recognition and display rendering.

I could care less if anyone writes code specific to certain benchmarks. Real world testing is where the proof is. Not in Anadtech's lab. This story is about as stupid as things get, which is probably why it is all over AppleInsider.

 

[MRU]

Basically further proof that synthetic benchmarks and end user experience are often a million miles apart...

 

[BaldiMac]

Wow. Like this really matters? Did not seem to matter to anyone before when Apple "lied" about the iPhone 5's CPU being twice as fast as the A5X in the 4S? Actual testing revealed, the iPhone 5 was neither twice as fast, nor faster than the 4S at all in many CPU/GPU aspects. In fact, the 5 was slower in things like speech recognition and display rendering.

Yes, there is a big difference between cheating a benchmark and your made up allegation of a "lie" by Apple.

I could care less if anyone writes code specific to certain benchmarks. Real world testing is where the proof is. Not in Anadtech's lab. This story is about as stupid as things get, which is probably why it is all over AppleInsider.

Yeah! Who cares if you cheat on benchmarks?! There are other ways to compare! **Unless they cheat on those too**

It's not like we shouldn't trust them just because they are being dishonest.

 

[lazard]

Yes, there is a big difference between cheating a benchmark and your made up allegation of a "lie" by Apple.



Yeah! Who cares if you cheat on benchmarks?! There are other ways to compare! **Unless they cheat on those too**

It's not like we shouldn't trust them just because they are being dishonest.

big deal. People tweak settings/overclock all the time to get optimized benchmark scores.

 

[ChazUK]

Some people take synthetic benchmarks seriously (hence the necessity for Samsung to game the system like they have).

If they weren't at all important, I doubt that "benchmark booster" code would even exist.

I've seen the same thing over the years from Nvidia, ATi, 3dfx and others so this is nothing new (but most of that was driver software tweaks rather than hardware overclocks).

It is funny to see the smartphone works of benchmarking emulating what PC component manufactures were doing years ago.

 

[MRU]

Some people take synthetic benchmarks seriously (hence the necessity for Samsung to game the system like they have).

.

Yep sadly this is very true...

How many youtube videos, blogs, hands on all judge a devices 'worthiness' via AnTuTu and Quadrant scores? Answer = a lot...... And despite evidence that these synthetic benchmarks aren't really the best way to judge end user experience.

they still use them, even on major review sites....

We so need some way of benchmarking, but these old legacy apps are probably not the best way and we need someone to do a complete rethink.

 

[ChazUK]

We so need some way of benchmarking, but these old legacy apps are probably not the best way and we need someone to do a complete rethink.

Definitely!

Sadly, lots of people like it back things up with arbitrary numbers that sum up to very little and benchmark scores are a good number to band about when they're on your side.

There have been plenty of heated discussions on these very boards over GLBenchmark scores and GPU's so I'm not too sure why people would be so dismissive over Samsung getting caught out like this.

Like I said, the very fact that Sammy went to the effort of doing this alone is quite telling IMO.

 

[Assault]

Yes, there is a big difference between cheating a benchmark and your made up allegation of a "lie" by Apple.



Yeah! Who cares if you cheat on benchmarks?! There are other ways to compare! **Unless they cheat on those too**

It's not like we shouldn't trust them just because they are being dishonest.

Really?

 

[Sensamic]

I always disliked Samsung, even though I have an SGS3.

I've only bought two products from them and don't plan on buying more in the future.

 

[lazard]

Some people take synthetic benchmarks seriously (hence the necessity for Samsung to game the system like they have).

If they weren't at all important, I doubt that "benchmark booster" code would even exist.

I've seen the same thing over the years from Nvidia, ATi, 3dfx and others so this is nothing new (but most of that was driver software tweaks rather than hardware overclocks).

It is funny to see the smartphone works of benchmarking emulating what PC component manufactures were doing years ago.

It's not only component manufacturers who do it. Just go look at the scores on benchmarks like FutureMark. Almost all the benchmark runs are done by regular Joes with overclocked CPUs/GPUs/RAM.

 

[Dr McKay]

Plenty of people cheat benchmarks. People have been known to put a phone in the freezer prior to running the benchmark etc, this is another level of cheating. It's why I don't pay much attention to benchmarking.

 

[Assault]

Definitely!

Sadly, lots of people like it back things up with arbitrary numbers that sum up to very little and benchmark scores are a good number to band about when they're on your side.

There have been plenty of heated discussions on these very boards over GLBenchmark scores and GPU's so I'm not too sure why people would be so dismissive over Samsung getting caught out like this.

Like I said, the very fact that Sammy went to the effort of doing this alone is quite telling IMO.

It doesn't matter because the S4 and all follow on Android flagship phones are, and will be, noticeably faster than any iphone (just the nature of Android, not a slam on Apple). There is no comparison being done here.
Not that benchmarks were accurate to begin with, and you certainly can't compare iOS and Android via benchmarks, because it is impossible to do so with any accuracy.

So why do we care again? Or is this just some way for the anti-Samsung crowd to try and throw crap at the internet wall? The only thing that should matter is how fast the phone is to you! That is the real benchmark.

 

[ChazUK]

It's not only component manufacturers who do it. Just go look at the scores on benchmarks like FutureMark. Almost all the benchmark runs are done by regular Joes with overclocked CPUs/GPUs/RAM.

Yes but I do feel it's different a single user tweaking their system for a higher 3DMark score versus the manufacturer actively lowering visual quality in drivers whenever they see a benchmark application is run (or in this case, to use higher than their maximum use level clock speeds for a higher benchmark score).

The guys and girls at Futuremark got wise to this which is why they have the approved drivers programme.

http://www.futuremark.com/support/approved-drivers

People rely on 3DMark to produce accurate results. We safeguard that trust by testing each vendor's drivers to ensure they do not unfairly influence or manipulate benchmark scores. You should only compare 3DMark scores when using the approved Windows drivers on this page. Older drivers may also have been approved but in general we recommend using the latest approved drivers to get the best results for your system.

The whole community element around Futuremark is pretty much about tweaking and maximising the performance out of a users PC.

 

[MRU]

.

So why do we care again? Or is this just some way for the anti-Samsung crowd to try and throw crap at the internet wall? The only thing that should matter is how fast the phone is to you! That is the real benchmark.

But....


1 input = if we don't care, but clearly Samsung do care enough when they

2 inputed code to enhance synthetic benchmark.

and if

3 = its an Anti-Samsung conspiracy and if real world benchmarks are the only thing that is important then

4 = return to 1

ad-nauseum



We do need some form of benchmarking. benchmarking itself gives us some way of measuring improvement of components, but clearly the current format is flawed and perhaps needs a rethink.

Oh and I wouldn't be surprised if all manufacturers (not just Samsung) are doing something similar to maximise AnTuTu and Quadrant scores somewhat artificially (or at least not fully representative).

 

[BaldiMac]

Really?

Really! Are you going to pretend the words "up to" aren't there?

 

[ChazUK]

There is no comparison being done here.
Not that benchmarks were accurate to begin with, and you certainly can't compare iOS and Android via benchmarks, because it is impossible to do so with any accuracy.

At what point did anyone bring the iPhone into this? This was never about Apple, HTC, Microsoft or anyone else in particular,

You may have been making the association because of some shoddy writing by someone at AppleInsider, I don't know, I don't read the crud they put out.

The big thing is that any Android OEM using the same SOC could be falsely perceived as putting out inferior hardware if two devices are benched with the same advertised base clock speeds which is what I personally think Samsung are up to here.

So why do we care again? Or is this just some way for the anti-Samsung crowd to try and throw crap at the internet wall? The only thing that should matter is how fast the phone is to you! That is the real benchmark

I'm not anti-Samsung at all either and have spent a good few pounds on their hardware (Samsung SSD 840, Galaxy S 1,2 & 3, Galaxy Nexus, Samsung Chrome book, Nexus S).

I definitely agree that real world usage is the only benchmark that should resonate with people but synthetic benchmarks do influence people and Samsung must know this.

 

[BaldiMac]

big deal. People tweak settings/overclock all the time to get optimized benchmark scores.

Ahh... The "other people do it" excuse. They cheated. Obviously it's a big deal to Samsung, or they wouldn't have put in the extra work to do it.

 

[ChazUK]

Oh and I wouldn't be surprised if all manufacturers (not just Samsung) are doing something similar to maximise AnTuTu and Quadrant scores somewhat artificially (or at least not fully representative).

How about Intel?
http://m.computing.co.uk/ctg/news/2282482/revised-benchmark-downgrades-intel-atom-against-arm

Although, this might not have been foul play from Intel.

 

[Stuntman06]

Samsung's biggest mistake here is that they got caught. "Benchmark Booster"? Really? They deserved to get caught. It's like planning to rob a bank and your car has the words, "Getaway Car" plastered all over.

No matter what benchmark system you use, people and companies will be able to find a way to artificially inflate these numbers. It's been done for PC's in the past, so I am not surprised it is happening now on mobile devices. These scores are important, so boosting them is important for companies. It's wrong to game the system, but I don't expect any company to not try doing it.

 

[lazard]

Samsung's biggest mistake here is that they got caught. "Benchmark Booster"? Really? They deserved to get caught. It's like planning to rob a bank and your car has the words, "Getaway Car" plastered all over.

No matter what benchmark system you use, people and companies will be able to find a way to artificially inflate these numbers. It's been done for PC's in the past, so I am not surprised it is happening now on mobile devices. These scores are important, so boosting them is important for companies. It's wrong to game the system, but I don't expect any company to not try doing it.

if you ain't cheatin you ain't tryin

 

[Fernandez21]

At what point did anyone bring the iPhone into this? This was never about Apple, HTC, Microsoft or anyone else in particular,

You may have been making the association because of some shoddy writing by someone at AppleInsider, I don't know, I don't read the crud they put out.

That seems to be the MO anytime anyone brings up Samsung in a negative light. People either start accusing Apple or apple fanboys or start bashing HTC, doesnt matter if either company was brought up originally. I dont know why people seem to take these things so personally.

 

[Shuri]

That seems to be the MO anytime anyone brings up Samsung in a negative light. People either start accusing Apple or apple fanboys or start bashing HTC, doesnt matter if either company was brought up originally. I dont know why people seem to take these things so personally.

Well that is pretty easy to explain.

Human beings (and monkeys) have very strong evolved Brains for two reasons; first one is scamming others and second one preventing from being scammed.

It's the second part mixing up with the misbelieve, that there has to be the ONE AND ONLY answer, proven by irrefutable facts, for every question.

So one of these Questions might be, which one is the best Smartphone maker around. And by stating, that there is only one possible answer, being true for everybody, we get two groups of people; those who buy those magical phones and those who don't.

Now because people don't want to feel betrayed, they must prove to themselves that they are in the first group of people, since they spend a lot of money, which requires to denounce the others and put them into the other group.

This is one of the things proving how stupid people really are...
And I don't tend to exclude myself of this stupidity.