Our New Testing Suite for 2018 and 2019

Spectre and Meltdown Hardened

In order to keep up to date with our testing, we have to update our software every so often to stay relevant. In our updates we typically implement the latest operating system, the latest patches, the latest software revisions, the newest graphics drivers, as well as add new tests or remove old ones. As regular readers will know, our CPU testing revolves an automated test suite, and depending on how the newest software works, the suite either needs to change, be updated, have tests removed, or be rewritten completely. Last time we did a full re-write, it took the best part of a month, including regression testing (testing older processors).

One of the key elements of our testing update for 2018 (and 2019) is the fact that our scripts and systems are designed to be hardened for Spectre and Meltdown. This means making sure that all of our BIOSes are updated with the latest microcode, and all the steps are in place with our operating system with updates. In this case we are using Windows 10 x64 Enterprise 1709 with April security updates which enforces Smeltdown (our combined name) mitigations. Uses might ask why we are not running Windows 10 x64 RS4, the latest major update – this is due to some new features which are giving uneven results. Rather than spend a few weeks learning to disable them, we’re going ahead with RS3 which has been widely used.

Our previous benchmark suite was split into several segments depending on how the test is usually perceived. Our new test suite follows similar lines, and we run the tests based on:

  • Power
  • Memory
  • Office
  • System
  • Render
  • Encoding
  • Web
  • Legacy
  • Integrated Gaming
  • CPU Gaming

Depending on the focus of the review, the order of these benchmarks might change, or some left out of the main review. All of our data will reside in our benchmark database, Bench, for which there is a new ‘CPU 2019’ section for all of our new tests.

Within each section, we will have the following tests:

Power

Our power tests consist of running a substantial workload for every thread in the system, and then probing the power registers on the chip to find out details such as core power, package power, DRAM power, IO power, and per-core power. This all depends on how much information is given by the manufacturer of the chip: sometimes a lot, sometimes not at all.

We are currently running POV-Ray as our main test for Power, as it seems to hit deep into the system and is very consistent. In order to limit the number of cores for power, we use an affinity mask driven from the command line.

Memory

These tests involve disabling all turbo modes in the system, forcing it to run at base frequency, and them implementing both a memory latency checker (Intel’s Memory Latency Checker works equally well for both platforms) and AIDA64 to probe cache bandwidth.

Office

  • Chromium Compile: Windows VC++ Compile of Chrome 56 (same as 2017)
  • PCMark10: Primary data will be the overview results – subtest results will be in Bench
  • 3DMark Physics: We test every physics sub-test for Bench, and report the major ones (new)
  • GeekBench4: By request (new)
  • SYSmark 2018: Recently released by BAPCo, currently automating it into our suite (new, when feasible)

System

  • Application Load: Time to load GIMP 2.10.4 (new)
  • FCAT: Time to process a 90 second ROTR 1440p recording (same as 2017)
  • 3D Particle Movement: Particle distribution test (same as 2017) – we also have AVX2 and AVX512 versions of this, which may be added later
  • Dolphin 5.0: Console emulation test (same as 2017)
  • DigiCortex: Sea Slug Brain simulation (same as 2017)
  • y-Cruncher v0.7.6: Pi calculation with optimized instruction sets for new CPUs (new)
  • Agisoft Photoscan 1.3.3: 2D image to 3D modelling tool (updated)

Render

  • Corona 1.3: Performance renderer for 3dsMax, Cinema4D (same as 2017)
  • Blender 2.79b: Render of bmw27 on CPU (updated to 2.79b)
  • LuxMark v3.1 C++ and OpenCL: Test of different rendering code paths (same as 2017)
  • POV-Ray 3.7.1: Built-in benchmark (updated)
  • CineBench R15: Older Cinema4D test, will likely remain in Bench (same as 2017)

Encoding

  • 7-zip 1805: Built-in benchmark (updated to v1805)
  • WinRAR 5.60b3: Compression test of directory with video and web files (updated to 5.60b3)
  • AES Encryption: In-memory AES performance. Slightly older test. (same as 2017)
  • Handbrake 1.1.0: Logitech C920 1080p60 input file, transcoded into three formats for streaming/storage:
    • 720p60, x264, 6000 kbps CBR, Fast, High Profile
    • 1080p60, x264, 3500 kbps CBR, Faster, Main Profile
    • 1080p60, HEVC, 3500 kbps VBR, Fast, 2-Pass Main Profile

Web

  • WebXPRT3: The latest WebXPRT test (updated)
  • WebXPRT15: Similar to 3, but slightly older. (same as 2017)
  • Speedometer2: Javascript Framework test (new)
  • Google Octane 2.0: Depreciated but popular web test (same as 2017)
  • Mozilla Kraken 1.1: Depreciated but popular web test (same as 2017)

Legacy (same as 2017)

  • 3DPM v1: Older version of 3DPM, very naïve code
  • x264 HD 3.0: Older transcode benchmark
  • Cinebench R11.5 and R10: Representative of different coding methodologies

Scale Up vs Scale Out: Benefits of Automation

One comment we get every now and again is that automation isn’t the best way of testing – there’s a higher barrier to entry, and it limits the tests that can be done. From our perspective, despite taking a little while to program properly (and get it right), automation means we can do several things:

  1. Guarantee consistent breaks between tests for cooldown to occur, rather than variable cooldown times based on ‘if I’m looking at the screen’
  2. It allows us to simultaneously test several systems at once. I currently run five systems in my office (limited by the number of 4K monitors, and space) which means we can process more hardware at the same time
  3. We can leave tests to run overnight, very useful for a deadline
  4. With a good enough script, tests can be added very easily

Our benchmark suite collates all the results and spits out data as the tests are running to a central storage platform, which I can probe mid-run to update data as it comes through. This also acts as a mental check in case any of the data might be abnormal.

We do have one major limitation, and that rests on the side of our gaming tests. We are running multiple tests through one Steam account, some of which (like GTA) are online only. As Steam only lets one system play on an account at once, our gaming script probes Steam’s own APIs to determine if we are ‘online’ or not, and to run offline tests until the account is free to be logged in on that system. Depending on the number of games we test that absolutely require online mode, it can be a bit of a bottleneck.

Benchmark Suite Updates

As always, we do take requests. It helps us understand the workloads that everyone is running and plan accordingly.

A side note on software packages: we have had requests for tests on software such as ANSYS, or other professional grade software. The downside of testing this software is licensing and scale. Most of these companies do not particularly care about us running tests, and state it’s not part of their goals. Others, like Agisoft, are more than willing to help. If you are involved in these software packages, the best way to see us benchmark them is to reach out. We have special versions of software for some of our tests, and if we can get something that works, and relevant to the audience, then we shouldn’t have too much difficulty adding it to the suite.

Test Bed and Setup CPU Performance: System Tests
Comments Locked

101 Comments

View All Comments

  • edzieba - Thursday, November 29, 2018 - link

    The difference is the ARM chips being labelled with the short-term term frequencies and performance, while Intel put the steady state values on the box. Motherboard manufacturers throw the box values right out the window, but if Intel were to dictate /those/ the wailing and gnashing of teeth from the peanut gallery would be cacaphonous.
  • melgross - Thursday, November 29, 2018 - link

    Its just a matter of semantics. It doesn’t have to be spelled out.
  • Targon - Thursday, November 29, 2018 - link

    There are actually three primary states. Base clocks, boost or turbo speeds, and then you can get thermal throttle which will actually lower the speed below the base clock speed. If the i9-9900k has a base of 3.6GHz, a turbo that goes up to 5GHz, but you have poor cooling, you may be seeing the CPU sticking to that 3.6GHz, or even below it if the temperatures get too high.

    This is where those very thin laptops may have Ryzen versions performing better than Intel, because of the temperatures keeping the chip running at or even below base speeds. For a small form factor machine, will the 9900k be running at base speeds ALL THE TIME due to temperatures/TDP/cooling? In the same small form factor case, would a Ryzen 7 2700X end up having a similar level of performance after several hours(to allow the heat generation to stabilize)? If you start when things are COLD, you could turn the machine on and run benchmarks, and see better numbers than if the machine were already on and you had been running intensive applications for several hours prior to running the benchmarks.
  • eastcoast_pete - Thursday, November 29, 2018 - link

    @Ian: Thanks for this informative test and review. One comment, one question/request. Comment: I continue to be struck by Intel's prowess when AVX512/AVX2 comes into play. I am also (negatively) impressed by the thermal load use of these instructions causes. The reduction in performance when using AVX512/AVX2 under strict adherence to a TdP of 95 Wh speaks volumes. Did you ever have a chance to ask Intel why running AVX makes their chips so power-hungry? Even if not, I'd appreciate your thoughts on why AVX makes Intel's chips run so hot.

    Here my question/request: I now that you/Anandtech have a large dataset on x264 video encoding speeds. However, especially for i7s and AMD's six-core and up Zen chips, I'd like to know how they fare when encoding/transcoding a 2160p 10bit video, as that is now in increasing demand, and really makes the processor sweat (and slow down, a lot). Any chance you and your colleagues can add that to the encoding tests? If space is an issue, I suggest to dump the x264 720p speed test; even a lowly Athlon or Celeron chip does that quite well, and at good speed.
  • HStewart - Thursday, November 29, 2018 - link

    I believe you can turn off AVX512 in bios - it use in special application that need the speed

    Also I would think the external GPU's is another major factor in considering power requirements on a system.

    I don't belkieve there is any power needs or reduction in topp speed for AVX2 only that AVX512 uses extra power on system and top frequency are reduce if being used.

    One thing about AVX2 - on Intel it is 256bit and AMD has dual 128 bits currently - not sure about new Zen's coming out next year. But at least with PowerDirector, it give you significantly performance increase
  • GreenReaper - Saturday, December 1, 2018 - link

    It's pretty simple, really: the more data the CPU has to process in parallel, the more horsepower it uses. It's like doubling or quadrupling the number of active cylinders in an engine - you gain performance, but it requires more power and produces more heat. That's why they're off if not in use.

    Dedicated GPU blocks for video coding will also use more power, but are likely to be far more efficient than doing the operations with general code - as long as it's within their defined capabilities. (Similarly, if you had to do the equivalent of the AVX operations without the relevant hardware, it would probably use even more power than it currently does, at least over the extra time it took.)
  • Davenreturns - Thursday, November 29, 2018 - link

    I have found much confusion among the readers on hardware review websites when it comes to this issue. So I would like to present some information from Anandtech's Bench tool in order to clarify the situation for me and others hopefully:

    Looking at the CPU Power Bench
    https://www.anandtech.com/bench/CPU-2019/2194

    The following two processors have these results under full package, full load:
    i7-6700k 82.55W First mainstream desktop 14 nm processor, 95W TDP according to Intel
    i9-9900k 168.48W Latest mainstream desktop 14 nm processor, 95W TDP according to Intel

    I assume that these two values were measured in unlimited mode. If this is the case, this means that the power listed above is when all cores/threads are loaded at full max turbo mode. So if you are expecting a certain level of performance given that Intel advertises 95W for both CPUs, then you are being misled and may not get the performance you are expecting when upgrading the CPU but not your cooling.

    This is a CHANGE from the past in how Intel uses TDP without telling the customer. It also highlights that Intel use to be conservative with cores/clocks/turbo when they had no competition and were able to shrink nodes between Nehalem and Skylake. Now they are PRETENDING that they can just double the cores and raise clocks on the same node and not increase power. Please correct me if I'm wrong, but it doesn't look like this is the case anymore.
  • AlyxSharkBite - Thursday, November 29, 2018 - link

    Really interesting how when you limit it to 95W it’s really close to the 2700X
  • 4800z - Thursday, November 29, 2018 - link

    A power unlimited 2700x. Also this article doesn't include any games. If it did you'd see the 9900k still does much better, because games don't use all 8 cores.
  • schujj07 - Thursday, November 29, 2018 - link

    2700X - 117.18W Max = 11.6% over stated TDP
    9900K - 168.48W Nax = 77.3% over stated TDP

    Don't forget not everyone views gaming and the end all be all form of benchmarking. Would it be interesting to see if it affects the gaming sure. It most certainly would affect those who game and stream at the same time.

Log in

Don't have an account? Sign up now