Adam Caudill

Security Leader, Researcher, Developer, Writer, & Photographer

On Automatic Updates and Supply Chain Attacks

Once again, a supply chain attack is in the news; this time, it’s a ransomware attack against Kaseya which has impacted hundreds if not thousands of businesses. According to Kevin Beaumont, the attackers used a 0day vulnerability in the Kaseya VSA appliance to deploy a fake update to all systems it managed; that update is actually the REvil ransomware. As this is a VSA is used by Managed Service Providers (MSPs), this resulted in an attack not just on the MSPs but also their customers.

This event raises an interesting question on update mechanisms and how they play into supply chain attacks. This is what we’ll be discussing today.

A History of Abuse #

Update mechanisms have been leveraged by attackers for many years and have, in some cases, had a substantial impact:

  • The Passwordstate attack used updates to insert malicious code that stole credentials.
  • The attack on SolarWinds pushed malware via updates to steal data from a variety of organizations, including the U.S. government.
  • NotPetya resulted in massive damage and was deployed via a backdoored version of M.E.Doc, once again delivered as an update.
  • Flame, perhaps the most famous example of this type of attack, leveraged the Windows Update system to deploy its malware.

These are but a few notable examples of incidents where an update mechanism is used to deploy some form of malware. Given this, one has to ask, are updates and especially auto-update mechanisms even safe to use?

Why Automatic Updates Exist #

Update mechanisms, automatic or otherwise, exists for a couple of primary reasons:

  • Ensuring that users are on a recent version, if not the latest. This allows developers to focus their efforts only on newer versions, which simplifies support, reduces the need to deal with compatibility, and gives users the latest and greatest features quickly.
  • Ensuring that users have the latest security fixes and improvements installed, to reduce the risk of being attacked by a known issue and provide additional protections.

It’s a well-understood fact that users aren’t good at manually installing updates, even when there are important security fixes. As such, there is a high likelihood that users will be rather behind the most recent version, and this is where automatic updates come in. By updating automatically, users don’t have to think about it, security teams don’t have to worry about it, and users get the best protection with the least effort.

The importance of updates should be clear to everyone, and the use of automatic updates plays an essential role in getting these updates deployed in a timely manner.

Security Updates Aren’t Just Fixes #

Before going on, I’d like to digress for a moment to address something that is too often overlooked. When most people think of a security update, they assume it fixes a vulnerability that was discovered — though that’s not always the case.

Updates may include additional hardening, which doesn’t address a known vulnerability, but instead adds extra protection to prevent exploitation or mitigate unknown vulnerabilities. This is more important than many people realize, as in some cases, these hardening improvements can block entire classes of vulnerabilities.

Such improvements don’t have a CVE, as they don’t represent a flaw, but without them, there may be an additional risk. Too often, these updates are skipped in manual update systems because it’s not clear enough what it does.

Accidental Fixes #

Another issue that is overlooked is how often vulnerabilities that are unknown or unreported are fixed unintentionally during refactoring or other code changes. Some updates may contain important security fixes; it’s just that the developer doesn’t know it.

One of the advantages of always using the latest version is that it increases the odds of these accidental fixes.

This fact may be surprising to those that don’t regularly work in vulnerability research, but it’s more common than one would expect.

The Importance of Securing Update Infrastructure #

To implement any update mechanism, and especially an auto-update mechanism, it’s vital that the infrastructure be hardened as much as possible, and the fewest number of people possible have access to it. There are few assets in a development shop more critical to protecting users than this infrastructure.

Servers must be locked down, psychical access restricted, code carefully reviewed, and operating system & software fully up to date.

Penetration testing should be conducted regularly, of not only the update infrastructure itself, but the entire system from end to end. For example, access to version control systems, build systems, signing systems, deployment & update systems, update notification, and update installation. Every step needs to be carefully reviewed and tested to ensure that there’s minimal risk of an attacker being able to influence what’s released.

Weighing the Risks #

The challenge now is this, how do the risks compare to the rewards? While this could be a case for a complete threat modeling exercise, I believe that there is an easier way to analyze this.

While some notable examples were listed above, the reality is that these attacks, while happening at an increasing pace, are still quite uncommon. After all, would we be having this discussion otherwise? Moreover, given how often auto-updating works, and the fact that with proper controls in place, the odds of a successful attack are small, there is a particularly strong argument that the rewards do indeed outweigh the risks involved.

While no system can achieve 100% security (give or take), we can achieve a high level of security by following the proper practices. It’s cutting corners and failing to properly invest in security that often leads to successful attacks against these mechanisms. While it’s true there’s risk, in reality, far more attacks are prevented than triggered.

Adam Caudill

Related Posts

  • Developers: Placing Trust in Strangers

    Much has been said, especially recently, about that mess of dependencies that modern applications have – and for those of us working in application security, there is good reason to be concerned about how these dependencies are being handled. While working on YAWAST, I was adding a new feature, and as a result, I needed a new dependency – ssllabs.rb. While most Ruby dependencies are delivered via Gems, ssllabs.rb is a little different – it pulls directly from Github:

  • Juniper, Backdoors, and Code Reviews

    Researchers are still working to understand the impact of the Juniper incident – the details of how the VPN traffic decryption backdoor are still not fully understood. That such devastating backdoors could make it in to such a security-critical product, and remain for years undetected has shocked many (and pushed many others deeper into their cynicism). There are though, some questions that are far more important in the long run:

  • Win by Building for Failure

    Systems fail; it doesn’t matter what the system is. Something will fail sooner or later. When you design a system, are you focused on the happy path, or are you building with the possibility of failure in mind? If you suffered a data breach tomorrow, what would the impact be? Does the system prevent loss by design, or does it just fall apart? Can you easily minimize loss and damage, or would an attacker have free rein once they get in?

  • Developers, Developers, Developers

    Note: This was written in 2012, but not published at the time. The point is still valid, perhaps moreso than ever and deserves to be made publicly. The content has been updated as appropriate, though the core of this article remains intact from the 2012 draft. I would like to note that this doesn’t apply to every environment, there are some where developers are very knowledgeable about security, and write code with minimal issues – my current employer happens to be one of those rare & exciting places.

  • Ruby + GCM Nonce Reuse: When your language sets you up to fail…

    A couple hours ago, Mike Santillana posted to oss-security about a rather interesting find in Ruby’s OpenSSL library; in this case, the flaw is subtle – so much so that it’s unlikely that anyone would notice it, and it’s a matter of a seemingly insignificant choice that determines if your code is affected. When performing AES-GCM encryption, if you set the key first, then the IV, and you are fine – set the IV first, you’re in trouble.