Analyzing Core i9-9900K Performance with Spectre and Meltdown Hardware Mitigations

One of the key aspects of the most recent Intel processor launch, its Core 9^th Generation processors, is that the new design affords some hardware-based protection for a couple of the Spectre and Meltdown family of security vulnerabilities. When these vulnerabilities were first discovered, they were patched using a combination of software and firmware, which unfortunately led to some performance regressions over an unpatched processor. The ultimate goal is for a hardware patch, which is always enabled, that loses zero performance – we’re testing out what the new patches have for us today.

A Recap: Spectre and Meltdown

At the start of 2018, it was announced that two new architecture-based attack vectors had been discovered in modern computer processors. These vectors are based in what are called side-channel attacks: the ability to probe or infer what something else on the processor is doing without actually seeing the workload or its data. But more fundamentally than that, the reason these attacks can occur is down to how the processor is designed. Side-channel attacks on processor speculation, for example, can expose data that wasn’t even intended to be accessed by the program in the first place. And other side-channel attacks can break through memory permission barriers between software like a Buick through a barn.

Because security is the number one factor in all modern day computing (and unfortunately in some cases, an afterthought), this became a serious potential problem quite quickly. The specific vulnerabilities have so far mostly fallen under two broad headings which are now synonymous with these attacks: Spectre and Meltdown. For completeness, there are similar attacks that don’t fall under these headings, but side-channel security is still at the heart of what is being broken.

All vulnerable processor makers were told about these attack vectors around six months before the public, in what is known in the industry as responsible disclosure – find a vulnerability, inform the vulnerable parties, and give them sufficient time to fix the issue before going public, in order to prevent any nefarious individuals exploiting it (and creating what is called a 0-day). In that time, the known attacks were analysed and software-related patches were built. For example, over the course of 2018, Intel started with its newest and high-priority platforms first, and slowly worked its way back through its product catalogue. Other companies had executed similar plans, however Intel has by far been the most high profile of them all.

One big issue surrounding these attacks is unsecured embedded systems running high-performance processors – those that are deployed in the field and can be easily compromised. This is beyond the scope of this article, but it’s important to understand how many devices might still be vulnerable in ten years.

How To Fix Spectre and Meltdown: Security

Since the disclosure of these new attack vectors, new vulnerabilities similar to them have been discovered but follow in the same vein. In order to protect against these issues, one extreme answer is a full top-to-bottom redesign of how modern computer processors are designed. This is a massive undertaking, so in the end we have a sliding scale of security patches.

Perhaps the easiest patch to implement is the operating system software patch. For home and office users, this is going to be the one that they experience first, given the prevalence of Windows and Mac OS to provide regular updates to the security of their platforms.

However, not everything can be solved by a traditional software patch, especially when the offending areas of the CPU design cannot be adjusted in at a high level software. More fundamental aspects of the CPU might require a microcode update, which involves uploading very small and low-level code to the CPU to make important changes to the CPU – and particularly to patch around errata.

The ideal way to distribute microcode updates is as part of a larger system firmware (e.g. BIOS) update, this way the updated microcode is available to the CPU immediately at boot. However it's also possible to distribute microcode updates at the OS level – to be loaded alongside the OS – and this is what has been happening given the serverity of the issue and the fact that OEMs tend to give up on firmware updates rather quickly. As a result, all the major OS vendors have been able to distribute the necessary microcode to affected CPUs through mainstream OS update mechanisms, allowing most PCs to be hardened against these attacks by combining the microcode fixes with OS changes to leverage the fixes.

But even as great as microcode is, it's still a software workaround for a hardware issue, incurring some performance penalties along the way. Ultimately there needs to be a hardware fix in place to ensure that the processor is secure at all times. This cannot happen on any current products, and requires new products over the next few years to have fixes in place.

Ultimately there are two types of hardware fix: a hardware patch or a hardware redesign. The hardware patch does the same as the software or microcode patches: stops the security issue. A hardware resign fundamentally changes the way the hardware operates, removing the issue entirely.

As stated, security is always a number one priority here. However priority one-point-five is performance. This is the reason for this review.

How To Fix Spectre and Meltdown: Performance

The reason why some of these new attack vectors have been so surprising is because they have arisen out of techniques designed to increase the performance of a modern day microprocessor. A key innovation to performance in the modern era is speculation – the ability for the processor to ‘predict’ (for lack of a better word) what data will be needed by the software at a future date, saving long trips out to main memory and hiding those delays. One of the slowest things you can do on a modern processor is main memory latency, so a lot of work has gone in to optimizing the hardware that performs speculation and pre-fetching data. As a result, speculative execution is one of the cornerstones of modern, high-performance processors.

The security problem arises when data is pre-fetched but not needed. If another program can access that data or infer about what the data contains, which may have been within a secure area in main memory, it’s a security nightmare. Or even another program can access that speculative memory even during the speculation itself, it’s a nasty thing to track. In an ideal world, everyone keeps to their own lane and everything runs fast and smooth. However, in order to keep that part of the memory secure, a fix needs to occur: either a patch, or a fundamental redesign of the hardware.

The patches for security issues such as speculation, therefore, have the potential to be dire for performance. If all of a sudden one of the techniques for increasing single core performance was disabled, that’s a lot of performance out of the window. Depending on whom you ask, the performance results of the initial software and firmware patches varied.

Speaking with Intel and their own internal performance numbers on server workloads, there were three main trends to speak of: most of the performance regression happened with memory and IO-related workloads (due to what is called kernel switching, one of the fixes applied), it has more of an effect with faster storage such as NVMe-based SSD (because HDDs already take a longer time to be ready), and it affects the newest processors the least. Going through the numbers on their tests for cloud servers, an IO/compute mix workload on modern hardware, such as HHVM Wordpress, would experience a 9-10% decrease in performance. An older platform such as Broadwell-EP or Haswell-EP performs similarly in all tests before and after patch, but may experience a 30-60% decrease an synthetic IO bound workloads such as FIO (worst case scenario) – Skylake-SP is less affected.

However, it was published in other media that some video game servers saw their game servers move from being 20% active to 50% active when the patches were applied. This increases costs and an inability to scale, so it has been suggested that some performance-limited deployments may not be using these fixes because of this. Others, such as compute-limited HPC, have seen no decrease in performance. Notably, it wasn’t even a topic at Supercomputing this year as to performance related issues.

The ultimate goal is to have a patch that minimizes the performance deficit, but keeps 100% of security. This has been erroneously reported in some press as the holy grail ‘hardware fix’, but those publications fail to differentiate between a hardware patch and a hardware redesign. A ‘simple’ hardware patch would still have many of the same performance issues as a software patch, just with the guarantee that said fixes are always there. But ultimately a hardware patch that minimizes performance loss is the goal.

Intel’s March to Hardware Fixes

If you have been following our coverage this year of Intel’s product portfolio, back in September it was announced that Intel’s next generation of server processors, Cascade Lake-SP, would implement a number of fixes at the hardware level. Lisa Spelman, VP of Intel’s Data Center Group, confirmed this in our interview, saying that the processor design hit the intercept for when the fixes were ready. Intel also stated that its Whiskey Lake 15W processors, built in the 14++ process, would have similar hardware fixes. And most recently, the new 9^th generation Coffee Lake Refresh process would also have the same fixes. As a result, we get a table like this of what is fixed and what is not:

So for the newest Core i9 Coffee Lake Refresh processors, CFL-R Intel has two hardware fixes, for Meltdown (aka Variant 3) and the closely related L1TF (aka Variant 5). All others will be covered by a mix of firmware and software patches.

As mentioned, the hardware fix is a minimum requirement for security, but says nothing about performance. The question then becomes if this fix is purely a patch, or something more. Out of all the vulnerabilities, Spectre Variant 2 has been quoted as the biggest source for performance regression.

Luckily, Intel has made this testing easy for us. The new 9^th generation Coffee Lake Refresh processors are microarchitecturally identical to the 8^th generation Coffee Lake family. As a result, we can take one processor from each, fix them to identical frequencies/core counts/memory, run them in the same motherboard with the same BIOS, and compare the benchmark results. This is what we did. If the hardware fix is more than just a patch, we should see the 9^th generation outperform the 8^th generation.

Test Setup

For our testing we took the newest Core i9-9900K and the special edition Core i7-8086K and initiated a face off. Each processor was set to four cores each with no hyperthreading, with the frequency set to 3.0 GHz for all workloads at all times. The motherboard of choice was our ASRock Z390 Pro Gaming Core i7, using BIOS P3.20, and we used 4x8GB of Crucial Ballistix DDR4-2666.

We ran through our whole benchmark suite, both our professional/office tests and our gaming tests using an MSI GTX 1080 at four different resolution/quality combinations. The results may shock you.

Results

Rather than several pages of graphs with two values in each, here’s a summary of the results split up by section.

In all cases, performance was within the margin of error between both processors. The biggest single CPU test gain was 4.0% (in LuxMark C++) and the worst was -2.7% (in Agisoft). The usual culprits for this sort of test, DigiCortex and WinRAR, were both within the margin of error. As a result, this hardware fix appears to essentially be a hardware implementation of the fixes already rolled out via microcode for the current Coffee Lake processors.

Cascade Lake-SP has similar fixes, but also has a hardware patch for Spectre Variant 2. In our interview with Lisa Spelman, she stated that Cascade overall will offer higher performance than Skylake-SP.

The long and short of matters then is that based on the testing we've done thus far, it doesn't look like Coffee Lake Refresh recovers any of the performance the original Coffee Lake loses from the Meltdown and Spectre fixes. Coffee Lake was always less impacted than older architectures, but whatever performance hit it took remains in the Refresh CPU design.

But at least the security for the two variants that Coffee Lake has hardware patches for is baked in, no matter where that processor goes. And Cascade will improve that further. It will be interesting to see where the next set of hardware fixes will land. If we can get some numbers, we’ll let you know.