By now you have likely heard the recent news of the Meltdown and Spectre vulnerabilities. Almost every computing system is affected by Spectre — desktops, laptops, smartphones, and servers. This brief will discuss Fortified Data’s guidance on the security risks. While we expect this guidance to be refined with additional information from the larger industry and our own testing, we are confident in the following observations and guidance.
Spectre & Meltdown
If you are not fully informed or aware of the recent vulnerability’s in chip architecture, this vulnerability includes processors from Intel, AMD, ARM and even Nvidia. Nearly every processor in circulation is impacted, however not all of them will have the same patches applied nor will all of them have the same performance penalty associated with them. One of the better articles describing these vulnerabilities can be found here.
Due to the differences in patching and the relative impact to varying workloads, the response to this issue is very complex. The simple answer is to patch everything. But realistically, our primary recommendation in this matter is to patch impacted systems and software as those updates become available, and are thoroughly tested, from both functional and performance aspects.
It’s important to note that the various patches by Intel, Microsoft, Oracle, anti-virus companies, etc., do not actually resolve the bug, but rather mitigate it or decrease the likelihood that the vulnerability can be exploited. This is not meant to be interpreted as a need to skip patches, but rather to test patches and apply them as appropriate. There have already been several well publicized patches which cause BSOD’s and / or limit functionality. They include: Anti-virus updates and Microsoft updates not working well together, Microsoft’s update for AMD processors and lastly SCCM functioning correctly with patches.
This newly found exploit applies to the majority of processors manufactured which are in current, active circulation, including, servers (on premise and those in the cloud), phones, and appliances (backup, storage).
Three primary issues exist: CVE-2018-5753, CVE-2017-5715 and CVE-2017-5754. Any one patch does not apply to all three of these vulnerabilities, so there will continue to be patches for the same exploit by different software and hardware companies. In this lies a good deal of complexity. For instance, it is possible that Microsoft will patch a vulnerability for Variant 2 while Intel patches the same vulnerability. This results in doubled performance implications. Should then one of these patches be rolled back? We are keeping a close eye on this unfolding story and will advise as necessary.
This table provides a high level summary of the exploits and their relevant CVEs:
|Exploited Vulnerability||CVE||Exploit Name||Public Vulnerability Name||Windows Changes||Silicon Microcode Update ALSO Required on Host|
|Spectre||2017-5753||Variant 1||Bounds Check Bypass||Compiler change; recompiled binaries now part of Windows Updates
|Spectre||2017-5715||Variant 2||Branch Target Injection||Calling new CPU instructions to eliminate branch speculation in risky situations||Yes|
|Meltdown||2017-5754||Variant 3||Rogue Data Cache Load||Isolate kernel and user mode page tables||No|
As time goes by it becomes difficult to determine the impact of a patch and which systems have their vulnerabilities addressed. To address this, we’ve analyzed our clients’ database environments and provided them with the relevant output. Please find “Speculation Control Settings.xlsx” attached to this e-mail. This attachment contains the output of the following script, modified by Fortified Data, but generally consistent with Microsoft’s original version.
There’s a significant amount of deviation in the larger industry on this topic. Many experts advise that everything should be patched, no questions asked. While we believe this to be a good default stance, there is a good case to not patch workloads where we are relatively certain untrusted code cannot run. Examples of this include certain storage appliances such as IBM and NetApp stating that their storage systems do not need to be patched.
For SQL Server, Microsoft provided patches for specific workloads on versions ranging from 2008 – 2017. It is important to note that Microsoft provides an outline regarding potential patching strategies and even states that in some scenarios, skipping the patch is a viable alternative. Oracle has begun to release patches and additional patches are expected over the next several weeks.
Based on our internal testing and observed workloads, we are confident that the following workloads will have a performance impact.
- High IO workloads
- Backup Appliances
- High Network workloads
- Backup Appliances
- Application Servers and Database Servers
The specific performance testing strategy will vary by client and by source system.
We will provide broad stroke analysis on specific workloads as we have time and availability to test them, however if you have applications with the two profiles noted above, we strongly recommend discussing Microsoft’s guidance on potentially weighing the value of untrusted code running on a specific server. We expect similar impact to Oracle workloads and any other workload with the aforementioned characteristics.
There have been a number of telling observations in the industry regarding the performance implication for mitigating these newly discovered vulnerabilities. The best example we can cite is based on our own, internal workload hosted in Azure. We run a fairly light database workload with 8 vCPU’s allocated. We will continue to test before/after scenarios and will update you with relevant information.
Not every workload will be impacted the same way. The older the processor and the older the O/S, the larger the impact. Systems which have PCID (Process Context ID) will have a lower performance implication when we enable PTI. Sandy Bridge and newer generations of processors will have a lower performance penalty than Westmere, Clarksfield or older generations of Intel processors.
To complicate this matter further, the performance impact is also relevant on the specific workload. Workloads with high disk activity and high network activity are more likely to have higher performance implications.
A couple of key examples of this can be found in the charts and links below.
The first is of our own, internal workload load mentioned previously. In Azure we are running a DS13_V2 with 8vCPU’s and 56GB of RAM.
As you can see, there’s been a marked increase in CPU time for the same workload in our hosted Azure environment.
For environments with high IO profiles, we expect to see significant performance degradation on writes.
We’ve created a perfmon template for Windows based workloads. This template should be run both before and after patching. It will measure the impact of your systems and provide data which will be useful in your internal conversations.
We intend to apply and begin running the perfmon template spectre_meltdownTemplate.xml. This same template can be used for any Microsoft O/S workload and is not targeted or specific for database workloads.
This is an involved and fluid situation which will take time and significant effort of those impacted. With most vulnerabilities, the solution to patch is relatively straight forward. Due to the inability to actually resolve this vulnerability and instead mitigate it as much as possible, it’s likely that we will continue to see updated patches as time passes.
The mitigation strategy is not as straightforward as one might hope because of the instability of some of these patches and the performance implication for certain workloads. It is imperative to communicate, test and stay up to date on this unfolding issue. Work with us to plan a thoughtful strategy and a cohesive response.
Fortified Data is here to assist, advise and continue to ensure the safety, reliability, availability and performance of your mission critical applications.
Further reading and details can be found in the URL’s below
|Acer||Acer only lists vulnerable desktop, notebook, and server products. Says it will release firmware updates for server products in March. No timeline for desktop and notebook products.|
|ASRock||The ASRock site is a mess. There’s no central security advisory, and users will have to visit the “Latest BIOS Update” page and sift through the updates by hand. The good news is that there are a lot of recent BIOS releases containing Intel microcode updates dated after the Meltdown and Spectre disclosure.|
|ASUS||ASUS says it will release BIOS updates for affected products by the end of January.|
|Dell||BIOS updates are available for some Dell desktop, notebook, and server products. The Dell security advisory contains several other links to various products types. You can use this page as the central hub to search for what you need.|
|Fujitsu||BIOS updates are available for some products, but not all. The security advisory contains multiple links to various product types.|
|Gigabyte||Motherboard provider Gigabyte has released BIOS updates. Users will have to access the advisory, click on the motherboard series name in the list of affected table, and check for a recent BIOS update on each motherboard product’s page.|
|HP||BIOS updates are available for almost half of the HP products listed as vulnerable.|
|Huawei||Huawei has only listed vulnerable products. Says an “investigation is still ongoing.”|
|Intel||Intel has released updates for most NUC, Compute Stick, and Compute Card products.|
|Lenovo||Lenovo has the best advisory yet, with detailed tables for all affected products, including download links and upcoming BIOS download availability for each one.|
|LG||There is no security advisory for the Meltdown and Spectre flaws. If readers spot one, please let us know in the comments.|
|Panasonic||Panasonic said it aims to release BIOS updates for vulnerable PC models starting the end of the month and continuing through February and March.|
|Microsoft||Microsoft has released UEFI updates for Surface products.|
|MSI||MSI has released BIOS updates.|
|Toshiba||Toshiba has not released any BIOS/UEFI updates just yet. The company lists affected products and an approximate timeline when it hopes to have updates available.|
|Vaio||Some BIOS updates are available. More to follow.|