Take
HTC's 10 as the most recent instructive example. It scored 88 out of 100 on DxOMark's mobile camera benchmark. If you're only reading the numbers, that rating makes the 10 equal to Samsung's
Galaxy S7 Edge, the only other smartphone to score as high as 88. I've used both extensively and I can tell you that's simply not the case — the S7 is outstanding whereas the 10 is merely good. Why do they then share the same score, and why is Sony's
Xperia Z5 sitting in third place with a no less impressive 87?
EMPIRICAL DATA CAN BE BEGUILING, AND OFTEN INCOMPLETE
The commonality between the 10, the S7, and the Z5 is in the excellent technical capabilities of their imaging sensors. But the divergence is in the actual results each camera produces and the experience of using the device. The Z5 is the best example of the limitations of benchmarks, because it
can take brilliant pictures, but most often doesn't because of its laggy performance. If your phone's software is so bad as to discourage you from even trying to snap a fleeting moment, what difference does the sensor's quality make?
Benchmarks, like statistics, exist to inform. We just need to be wary about misinterpreting their numerical certitude for factual completeness. 3DMark and DxO endeavor to cover most usage scenarios, but they'll never cover all, and part of the dissatisfaction with graphics performance benchmarks stems from their inadequate reflection of actual in-game performance.
If a laptop or a phone does well in a web-browsing battery benchmark, that only gives an indication that it would probably fare decently when handling bigger workloads too. But not always. My good friend Anand Shimpi, formerly of
AnandTech, once articulated this very well by pointing out how the MacBook Pro had better battery life than the MacBook Air — which was hailed as the endurance champ — when the use changed to consistently heavy workloads. The Pro was more efficient in that scenario, but most battery tests aren't sophisticated or dynamic enough to account for that nuance. It takes a person running multiple tests, analyzing the data, and adding context and understanding to achieve the highest degree of certainty.
