The systemic problem of Android benchmark cheating
Android’s openness at work again. Result – open wasteland full of cheating slime-balls.
Recently, Samsung was discovered to fake Note 3 & Galaxy S4 performance benchmarks. So, Anand Lal Shimpi & Brian Klug of Anandtech investigated other Android mobile devices. They discovered the state of mobile device benchmark cheating:
“We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop. With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.”
Just about all Android makers added custom code to increase device performance when running benchmarks apps. The faked performance is not available during daily use. The cheaters include: Asus, HTC, LG, and Samsung. I’ve colorized the table:
Android Benchmark Cheating
Instead of fixing their cheating problem, Android device makers such as Samsung appears to be putting more effort into hiding the code:
“Don’t forget that we’re lucky to be able to so quickly catch these things. After our piece in July I figured one of two things would happen: 1) the optimizations would stop, or 2) they would become more difficult to figure out. At least in the near term, it seems to be the latter. The framework for controlling all of this has changed a bit, and I suspect it’ll grow even more obfuscated in the future.”
Apple does not cheat benchmark scores.
- Benchmark: iPad 2 Faster than Android Xoom
- Samsung fakes Note 3 device performance benchmarks
- Performance Benchmarks: iPad 2 Still Rules Android, Samsung Galaxy
- Android Maker Motorola Mobility Ships 440,000 Xoom Tablet, Suffers 2011 Q2 Loss
- Android apps including those in Google Play Store vulnerable to MitM remote code execution exploit