A Message About Intel Security Findings
Update Friday, January 19th, 2018:
Over the past week, our engineering team has identified and begun formal testing on a set of kernel patches that begin to mitigate all three variants. We are validating this candidate kernel with all of our major subsystems and starting to plan and test this initial round of fixes to deploy across our fleet.
We anticipate a robust testing phase over the next week to ensure these changes will not negatively impact our customers, continuing our approach of taking careful, well-informed steps towards long term resolution, rather than a string of one-off mitigation efforts.
We plan to provide our next update on Friday, January 26th. If we determine that we are able to initiate reboots sooner, we will provide an update here and e-mail affected customers directly with at least 24 hours advance notice.
In the meantime, we encourage you to ensure your servers are as secure as possible. For more information about protecting your Droplets, you can reference this tutorial.
Meltdown distribution updates to date: CentOS 6 & 7, RancherOS, CoreOS, Debian 7, 8 & 9, Fedora 27 Atomic, Ubuntu 14.04, 16.04 & 17.10 base images have been updated. Fedora 26 & 27 have updates available, but users will have to manually update as Fedora does not have updated Cloud Images. Our Ubuntu 16.04 1-Click images have been refreshed to include the latest patches and updates.
Spectre distribution updates to date: At this time only CentOS 6 and CentOS 7 have released kernel updates to address Spectre Variant 1 and have been included in our our CentOS 6 & 7 base images.
Update Friday, January 12th, 2018:
Our engineering team continues to procure and test patches as they become available and we have a significant amount of resources dedicated to this task. While numerous patches have been rolled out for Meltdown, mitigations for Spectre are still sparse and raw. Unfortunately, many distributions have not been able to roll out a full set of patches to address all 3 variants of the exploits. We will continue preliminary testing through the week of January 15th. These tests will have no customer impact, and will be focused on getting machines up and running in these new environments.
Intel released a microcode update this week, unfortunately the update was determined to cause stability issues for other Intel customers, and has since been pulled back. DigitalOcean did not apply this microcode to our fleet, and we are awaiting the release of new a microcode. Once we have the final microcode in hand, we will begin performance/regression testing to validate the update in our environment.
As we have previously mentioned, fleet wide reboots will take place following successful testing and validation. We will communicate the reboot schedule to customers in advance of any action. In the meantime, we expect to share another update here on Friday 1/19. For more information about protecting your Droplets, you can reference this tutorial.
Meltdown distribution updates to date: CentOS 6 & 7, RancherOS, CoreOS, Debian 7, 8 & 9, Fedora 27 Atomic, Ubuntu 14.04, 16.04 & 17.10 base images have been updated. Fedora 26 & 27 have updates available, but users will have to manually update as Fedora does not have updated Cloud Images.
Update Tuesday, January 9th, 2018:
As the ongoing security vulnerability developments evolve there are still many unknowns. Like many other cloud service providers we are participating in Linux kernel working groups, coordinating with Intel and other hardware vendors, and doing our own exhaustive research. The goal is to protect the security of our users' data and provide a long term solution instead of offering a cascade of short term fixes. That said, here is our approach based on what we know today.
As mitigations for vulnerabilities are released, our engineering team is diligently and methodically testing each one to ensure that our customers have stability and performance when the patches are applied. We will continue this testing process for all new patches. It is difficult to estimate the timeframe we’ll need to create, debug and test them, as new patches are being rolled out sporadically, but we anticipate that the testing phase will last for at least another week. We plan to share another update this Friday, January 12th. As mentioned in our last post, we will alert customers in advance of any reboots that need to take place.
In the meantime, we encourage you to track the patches being released on your distributions and we’ve compiled a list of distribution patches released thus far, which we will update as they become available. It’s important to note that updated distributions do have various fixes, but none have remediations for all three vulnerabilities. In order to help our users protect themselves as the patches become available, we have changed all Droplets to utilize our Grubloader kernel, ensuring that Droplet kernels can be upgraded by the user, without DigitalOcean involvement.
Meltdown distribution updates to date: CentOS 7, RancherOS, CoreOS, Debian 9, Fedora 27 Atomic
We’ve also authored this tutorial to help you apply patches. This, too, will be updated as more information and patch releases become available.
Update Friday, January 5th, 2018:
Our engineering team continues to remain in close coordination with Intel, Canonical, and our other vendors. We are currently awaiting patches that, once applied, should mitigate the security vulnerabilities. We expect to have those patches on Tuesday, January 9th, and will begin formal testing as soon as they are received.
In the interim, as patches become available on the Linux kernel list and microcode updates become available from Intel and other vendors, we are doing ad-hoc testing to understand potential performance implications and evaluate stability concerns so we can execute our mitigation as smoothly as possible.
The scope of work is extensive; everything from the kernel to compilers and emulation systems must be patched and tested. We will be devoting all of our available engineering resources to this effort, but the set of changes is so significant that we cannot yet estimate the time frame needed to complete validation. The security of our customers and the reliability of our services are important to us and it is critical that we take the time to validate before we roll out changes.
We expect to post another update on Tuesday, January 9th, once we have received the patches and testing is underway. We will share updates here sooner if additional information becomes available. We appreciate your patience!
Update Thursday, January 4th, 2018:
Our engineering team continues to coordinate closely with Intel to determine the exact scope and impact of the Meltdown and Spectre security vulnerabilities. It is our current understanding that DigitalOcean is not vulnerable to the Meltdown (Variant 3) exploit because of our usage of KVM virtualization. However, we will still be taking the necessary steps to protect our customers from the impact of the Spectre (Variants 1 and 2) exploits.
We will be obtaining the patches necessary to mitigate the vulnerabilities and once our engineering team has validated them, we will be rebooting our entire fleet of Droplets. DigitalOcean users will also need to upgrade their own kernels, and we will be working closely with them to ensure that this process goes as smoothly as possible. Every customer will receive advanced notification before we initiate the reboots.
Original post Wednesday, January 3, 2018:
Earlier this week, we became aware of a potential security flaw impacting Intel hardware used by DigitalOcean and many other cloud service providers. Since learning of this issue, we have been actively investigating and tracking Linux kernel activity and our development team has been working diligently to obtain as much information as possible from Intel. Unfortunately, the strict embargo placed by Intel has significantly limited our ability to establish a comprehensive understanding of the potential impact.
Based on our investigation and the information we have received thus far, we believe that it may be necessary to reboot impacted customer Droplets. If reboots are determined to be the correct course of action for DigitalOcean users, we will schedule the urgent maintenance and notify impacted customers in advance.
We are continuing to monitor this situation and work with Intel to obtain more details. We’ll share updates in this blog post as additional information becomes available to us.
You can read Intel’s initial statement here.
Josh Feinblum leads security and compliance for DigitalOcean and serves as Chief Security Officer. Prior to DigitalOcean, he was the head of security at Rapid7 and started several security programs across hyper-growth, technology-oriented healthcare companies. He is deeply involved in the security community and has more than 14 years of experience managing security teams, overseeing major clients at large managed service providers, and starting privacy and security related programs across commercial and federal financial service firms.