Post-Advisory Exposure

James Wetter and Nicky Ringland, Open Source Insights Team

How can a user of open source software (OSS) assess their risk of exposure to a future vulnerability when taking on a new dependency?

Vulnerabilities will always find their way into software, and in an ideal world those vulnerabilities will be fixed in a reasonable amount of time. This is a critical factor for building trust between OSS maintainers and the users of their software.

This blog post looks at the events around the remediation of a vulnerability, and a few ways that trust can be established between maintainers and users of OSS. In particular we examine how often OSS packages remediate known vulnerabilities and if their users were left exposed after the vulnerability was publicly disclosed.

An ideal remediation

next-auth is an npm package that provides tools to help implement authentication for the web development framework Next.js. next-auth is popular, with almost 200,000 weekly downloads according to npm. Recently an advisory was published detailing a critical vulnerability in the next-auth package. Due to this vulnerability, an attacker could potentially gain access to another user’s account.

Fortunately for the users of next-auth, the reporter of the vulnerability and package maintainer practiced coordinated vulnerability disclosure. As a result a fixed version of next-auth was already available when this advisory was published. Both versions 4.10.3 and 3.29.10 include a patch remediating the vulnerability.

The advisory itself contains a brief timeline of key events. The vulnerability was discovered by Socket, and privately disclosed to the maintainers of next-auth on the 26th of July. The maintainers acknowledged the private disclosure within 1 hour, and released remediating versions on the 1st of August. Two days later, an advisory disclosing the vulnerability was published. The time between private disclosure and the release of a fix, the time to remediation, was approximately 5 working days.

The events surrounding a coordinated vulnerability disclosure.
The events surrounding a coordinated vulnerability disclosure.

This situation is ideal. Both the private disclosure of the vulnerability and rapid response of the package maintainers meant that the two most recent major versions both had patched versions available for users before the publication of the advisory.

By the time the advisory was published, most users of the next-auth package would be able to move to a patched version immediately with little effort. This virtually eliminated the post-advisory exposure time for the many users of the package.

What can go wrong?

Things don’t always work out as well as this, though. There are a few ways in which the process could go awry such as the discovery of a zero-day exploit, or a vulnerability in an unmaintained package.

A zero-day exploit

The events surrounding a hypothetical zero day exploit.
The events surrounding a hypothetical zero day exploit.

A zero-day exploit is when a vulnerability is being actively exploited by the time the package maintainers become aware of the issue. In these situations it may be better to publish an advisory before the maintainers have developed a patch in order to raise awareness as quickly as possible. This was the case for the well publicized remote code injection vulnerability in the popular log4j library.

In this scenario, it is not reasonable to expect the maintainers to remediate the vulnerability before the advisory is published - increased awareness is a higher priority. And as a result the users of the package will be exposed to a publicly known vulnerability until a remediation is made available, or they remove their dependency on the affected package.

An unmaintained package

The events surrounding a failed coordinated vulnerability disclosure for a hypothetical unmaintained package.
The events surrounding a failed coordinated vulnerability disclosure for a hypothetical unmaintained package.

When a vulnerability is discovered in a package that is no longer maintained there will be no response to private disclosure, leaving the reporter no choice but for the reporter to publish an advisory without a fix available.

An example of this situation is the once popular npm package parsejson. Its most recent release has an unremedied, high severity vulnerability that was publicly disclosed in 2018. But the package hasn’t seen a new release since 2016. Its GitHub repository has been archived and clearly states that it is no longer maintained. Worryingly, the package is still widely used: npm reports that the package still gets almost 250,000 weekly downloads.

It’s clear that users of OSS should not introduce new dependencies on an unmaintained package like parsejson. Existing users should remove such dependencies from their libraries and applications as quickly as they can. But it can be hard for a developer to know when one of their dependencies is no longer maintained or less actively maintained. Signals to help identify changes in the maintenance status are critical.

What usually happens after an advisory?

For our discussion here, we consider a package to have remediated an advisory when it has a release that

  1. is not affected by the advisory, and
  2. has a greater version number than all affected releases.

The semantics of versions and release differ between systems. For example PyPI uses pep440, while npm uses semantic versioning.

This definition of remediation means that if the greatest major version of a package has a fix available, the package is considered to have remediated the vulnerability even if lesser major versions remain affected. There is more to be said about packages that have multiple major versions, each of which may be fixed independently, but we will leave a discussion of the nuance of vulnerabilities and multiple major versions for another time.

Clearance rates

First let’s take a look at how many known vulnerabilities are remediated.

Across every package management system supported by deps.dev, we see that most package maintainers do respond to vulnerability advisories in their packages.

There is considerable variation between ecosystems. The lower clearance rate seen in the Cargo ecosystem is expected. Within that ecosystem, there is a practice of publishing an advisory that a package is unmaintained, such as this advisory and this advisory. Such advisories are not expected to be remediated, but publishing them helps raise awareness of the package’s unmaintained state amongst its users.

Taking a closer look at individual packages, the clearance rate of vulnerabilities gives an indication of the health of the package, and consequent risk of using the package. Some packages have a very high number of known vulnerabilities in older versions, but all of the vulnerabilities have been remediated. For example

These packages are healthy and well maintained, and their high clearance rates are a good indication of that.

Post-advisory exposure time

Now let’s consider how long users are exposed to a known vulnerability without a fix. That is, the interval between the publication of an advisory and the publication of a release to remediate it. We call this the post-advisory exposure time.

The PyPI, Cargo and npm packaging systems expose the publication times for each version. Using this data we can examine the post-advisory exposure time.

The post-advisory exposure time across the PyPI, Cargo and npm ecosystems for all the vulnerabilities in the deps.dev dataset.
The post-advisory exposure time across the PyPI, Cargo and npm ecosystems for all the vulnerabilities in the deps.dev dataset.

At a glance these graphs paint a positive picture. Each ecosystem appears healthy, with the majority of vulnerabilities disclosed in an advisory being remediated very quickly. This demonstrates that security is a priority for most maintainers.

But it should be noted that vulnerabilities where coordinated disclosure was successful will have zero post-advisory exposure time (or even negative time!). In npm and PyPI almost 60% of the vulnerabilities in our database were remediated before the publication of the corresponding advisory. Cargo has a much lower percentage, around 16%; more on that shortly.

Let’s direct our attention to cases that did not see a coordinated vulnerability disclosure. The following histograms show the post-advisory exposure time, excluding successfully coordinated disclosures.

The post-advisory exposure time across the PyPI, Cargo and npm ecosystems for the vulnerabilities in the deps.dev dataset that didn't have a coordinated disclosure.
The post-advisory exposure time across the PyPI, Cargo and npm ecosystems for the vulnerabilities in the deps.dev dataset that didn't have a coordinated disclosure.

In all three systems, many vulnerabilities are remediated within 30 days of advisory publication. This includes many zero-day exploits, such as log4shell, that were fixed as quickly as possible, even without the more ideal option of a coordinated vulnerability disclosure.

In the case of Cargo, the number resolved in the first 30 days is a staggering 70% of all vulnerabilities remediated after advisory publication. This is because many maintainers choose to release the remediation on the same day the advisory is published, resulting in non-zero but very brief post-advisory exposure time.

The long tail of vulnerabilities with significant post-advisory exposure time is a valuable signal on the health of the corresponding packages. For developers taking on new dependencies, knowing that they will not be left exposed for long periods of time is critical to their security posture. For existing users of a dependency, being aware of changes to future remediation likelihood of potential vulnerabilities is equally important.

Currently it is hard to know how a given package has previously performed according to this metric. Ideally this information would be easily accessible, allowing potential and existing users to make informed decisions about their dependencies.

Mean time to remediation

The number of known vulnerabilities that a package maintainer has remediated in the past can be used to help build trust between maintainers and users of OSS. Additionally, the length of time users of a package were left exposed to known, unremedied vulnerabilities in the past can provide a more detailed characterization of a package maintainer’s response.

In addition to these signals, Mean Time to Remediation (MTTR) has been proposed as a useful indicator of the quality of a package’s maintenance.

However, the available data about advisories rarely contains timestamps for critical events in the remediation process. For example, most advisory databases, including GitHub Advisories and OSV, do not provide a timestamp field for the private disclosure of the vulnerability or the maintainers acknowledgement. And while some advisory write-ups do include an event timeline, these are quite rare.

These missing timestamps make it impossible to compute the time that elapsed between a maintainer being notified of a vulnerability and the release of a remediation, relegating MTTR to a, for now, still hypothetical metric to compute.

Conclusion

Vulnerabilities are an inevitable part of software development. The code reuse and efficiency gains made possible by OSS broadens the potential impact of vulnerabilities.

But cooperation between parties that discover vulnerabilities and package maintainers reduces the time that users are left exposed to publicly known vulnerabilities. Thanks to the hard work of OSS maintainers, there is no post-advisory exposure for the majority of vulnerabilities in our advisory database.

Developers should still prepare for less ideal outcomes. Every dependency they introduce increases the risk of exposure to future vulnerabilities. The clearance rate and post-advisory exposure time for past advisories can provide users of OSS assurance about the quality of maintenance their dependencies receive. While past performance may not always predict future behavior, it can be used as a valuable signal to help make informed decisions.

All blog posts