UL Delists Huawei Devices Due To Inaccurate Benchmarks
by Andrei Frumusanu on September 6, 2018 12:55 PM EST- Posted in
- Mobile
- Futuremark
- Smartphones
- 3DMark
- UL
A few days ago we published our article addressing Huawei and Honor’s inaccurate benchmarking behaviour. In a nutshell, we had found out that this year’s devices had introduced a new thermal limiting behaviour that quickly throttled power consumption to ~4-4.5W in 3D workloads. While this in itself wasn’t an issue, the problem is that the firmware did not apply this new behaviour to a specific list of whitelisted 3D benchmarks.
Sidenote: Earlier this year, Futuremark (The 3DMark creators) had been acquired by UL, joining the global safety consulting and certification company as a benchmarking division.
We work closely with all benchmark vendors, and UL isn’t an exception. We had first given UL note of the behaviour two weeks ago and had been sharing our early results with the development team. Today UL published their independent confirmation of our results on their own devices, and have subsequently decided to remove the tested devices from their results database.
UL stresses that the kind of detection and optimization performed by Huawei infringes the company’s rules for manufacturers. Unfortunately this isn’t the first instance of a vendor being delisted, as most famously a slew of phone manufacturer had been delisted in 2013 after a more in-depth investigation of ours resulted in quite embarrassing results for a lot of vendors.
The UL team further explains that they’ve also been in touch with Huawei, and that the Chinese vendor is planning to address the behaviour by introducing a new “performance mode” that disables the new thermal throttling behaviour. In essence this mode would revert back to the behaviour we’ve seen in the past such as the Mate 9 – where the SoC is allowed higher peak performance figures at a cost of high power.
While this is still problematic for representative benchmarking for devices with Kirin 960’s and 970’s (due to the very large peak power at the highest performance states of those SoCs), I expect this to be a non-issue for the new Kirin 980, as it’s projected to have significantly better sustainable peak power figures. We're still looking forward to the new SoC in the upcoming Mate 20, and believe it to be an outstanding performer.
Source: UL News Release
Facebook
Twitter
RSS
33 Comments
View All Comments
HStewart - Thursday, September 6, 2018 - link
That is typical in sites with comments - look at the Intel article for example and other case like NVidia.HStewart - Thursday, September 6, 2018 - link
But it still better than say WCCFTech where comments are not required to be even register - and you have people showing game bench marks and other stuff - not even discussing subject.Personally I stay away from opposite views of subject - except when they complain about of other products in relationship with product in review. For example I don't normally discuss AMD issues unless some one states a comparison to what I have for example Intel and also any concerning drivers since I have Dell XPS 15 2in1 which has AMD GPU in it. Or possibly direct curious it on the product it self.
As for issue of bench marks, personally I feel it is extremely hard to bench mark - x86 vs ARM cpus especially phones. Difference is OS and Chips and such. It hard enough truly benchmark Intel vs AMD on x86 - especially that each has different instruction difference - and how much benchmark takes advantage of each platform. Plus of course single core speed vs Multi core speed is another part - with multi core - what is assurance that test is real world today when most of user interface is single threaded.
PeachNCream - Thursday, September 6, 2018 - link
TL;DR - "I'm a brand loyalist living in denial about my brand loyalty and I don't care that the brand I love is way off topic in these discussions." -HStewartHStewart - Thursday, September 6, 2018 - link
"TL;DR - "I'm a brand loyalist living in denial about my brand loyalty and I don't care that the brand I love is way off topic in these discussions." -HStewart"Please don't quote stuff that I never said. Just because you disagree with what I state, does not mean that you can lie about what I said.
I never said that - just that I am different then others - my taste has change over the years
Before I always prefer NVidia Graphics and Apple Phone
Now I see only the need to discreet graphics like my AMD cpu and have Samsung Phone.
Just I don't care about AMD has my CPU - they just don't fit my needs - too big and not where I desired to go. These comments are not off topic on the original thread - it just describes my opinion on why conversation on forums always go off topic on thread., Actually the last part of my original statement is on topic - bench marks are not accurate between different types of hardware.
HStewart - Thursday, September 6, 2018 - link
Comment correction since we can't edit: "my AMD cpu" should me "my Dell XPS 15 2in1 with AMD gpu"PeachNCream - Friday, September 7, 2018 - link
TL;DR - "I'm offended for being called out! Here's some distracting smoke and mirrors as damage control!" -HStewartgrahad - Monday, September 10, 2018 - link
He hasn't commented that he found a design bug in a CPU eons ago yet though WOW.Manch - Friday, September 7, 2018 - link
LOL, dude complains about people getting off topic...goes off topic to shill for Intel. Oh hey HStewart!!! Thought that was you!! BWAHAHAHAAA!!porcupineLTD - Friday, September 7, 2018 - link
I may be wrong but there was some other Intel shill that stopped posting around the same time HSteward started posting regularly, seems to be related.Manch - Monday, September 10, 2018 - link
Youre not wrong lol