How can a user of open source software (OSS) assess their risk of exposure to a future vulnerability when taking on a new dependency?
Vulnerabilities will always find their way into software, and in an ideal world those vulnerabilities will be fixed in a reasonable amount of time. This is a critical factor for building trust between OSS maintainers and the users of their software.
This blog post looks at the events around the remediation of a vulnerability, and a few ways that trust can be established between maintainers and users of OSS. In particular we examine how often OSS packages remediate known vulnerabilities and if their users were left exposed after the vulnerability was publicly disclosed.
An ideal remediation
next-auth is an npm package that provides
tools to help implement authentication for the web development framework
next-auth is popular, with almost 200,000
weekly downloads according to npm. Recently an
was published detailing a critical vulnerability in the
next-auth package. Due
to this vulnerability, an attacker could potentially gain access to another
Fortunately for the users of
next-auth, the reporter of the vulnerability and
package maintainer practiced coordinated vulnerability
a result a fixed version of
next-auth was already available when this advisory
was published. Both versions 4.10.3 and 3.29.10 include a patch remediating the
The advisory itself contains a brief timeline of key events. The vulnerability
was discovered by Socket, and privately disclosed to the
next-auth on the 26th of July. The maintainers acknowledged the
private disclosure within 1 hour, and released remediating versions on the 1st
of August. Two days later, an advisory disclosing the vulnerability was
published. The time between private disclosure and the release of a fix, the
time to remediation, was approximately 5 working days.
This situation is ideal. Both the private disclosure of the vulnerability and rapid response of the package maintainers meant that the two most recent major versions both had patched versions available for users before the publication of the advisory.
By the time the advisory was published, most users of the
would be able to move to a patched version immediately with little effort. This
virtually eliminated the post-advisory exposure time for the many users of the
What can go wrong?
Things don’t always work out as well as this, though. There are a few ways in which the process could go awry such as the discovery of a zero-day exploit, or a vulnerability in an unmaintained package.
A zero-day exploit
A zero-day exploit is when a vulnerability is being actively exploited by the
time the package maintainers become aware of the issue. In these situations it
may be better to publish an advisory before the maintainers have developed a
patch in order to raise awareness as quickly as possible. This was the case for
the well publicized remote code injection
vulnerability in the
In this scenario, it is not reasonable to expect the maintainers to remediate the vulnerability before the advisory is published - increased awareness is a higher priority. And as a result the users of the package will be exposed to a publicly known vulnerability until a remediation is made available, or they remove their dependency on the affected package.
An unmaintained package
When a vulnerability is discovered in a package that is no longer maintained there will be no response to private disclosure, leaving the reporter no choice but for the reporter to publish an advisory without a fix available.
An example of this situation is the once popular npm package
parsejson. Its most recent release has an
unremedied, high severity
vulnerability that was
publicly disclosed in 2018. But the package hasn’t seen a new release since 2016.
Its GitHub repository has been archived
and clearly states that it is no longer maintained. Worryingly, the package is
still widely used: npm reports that the package still gets almost 250,000 weekly
It’s clear that users of OSS should not introduce new dependencies on an
unmaintained package like
parsejson. Existing users should remove such
dependencies from their libraries and applications as quickly as they can. But
it can be hard for a developer to know when one of their dependencies is no
longer maintained or less actively maintained. Signals to help identify changes
in the maintenance status are critical.
What usually happens after an advisory?
For our discussion here, we consider a package to have remediated an advisory when it has a release that
- is not affected by the advisory, and
- has a greater version number than all affected releases.
This definition of remediation means that if the greatest major version of a package has a fix available, the package is considered to have remediated the vulnerability even if lesser major versions remain affected. There is more to be said about packages that have multiple major versions, each of which may be fixed independently, but we will leave a discussion of the nuance of vulnerabilities and multiple major versions for another time.
First let’s take a look at how many known vulnerabilities are remediated.
Across every package management system supported by deps.dev, we see that most package maintainers do respond to vulnerability advisories in their packages.
There is considerable variation between ecosystems. The lower clearance rate seen in the Cargo ecosystem is expected. Within that ecosystem, there is a practice of publishing an advisory that a package is unmaintained, such as this advisory and this advisory. Such advisories are not expected to be remediated, but publishing them helps raise awareness of the package’s unmaintained state amongst its users.
Taking a closer look at individual packages, the clearance rate of vulnerabilities gives an indication of the health of the package, and consequent risk of using the package. Some packages have a very high number of known vulnerabilities in older versions, but all of the vulnerabilities have been remediated. For example
tensorflow-cpueach with around 570 remediated advisories, and
jenkins-corewith 112 remediated advisories.
These packages are healthy and well maintained, and their high clearance rates are a good indication of that.
Post-advisory exposure time
Now let’s consider how long users are exposed to a known vulnerability without a fix. That is, the interval between the publication of an advisory and the publication of a release to remediate it. We call this the post-advisory exposure time.
The PyPI, Cargo and npm packaging systems expose the publication times for each version. Using this data we can examine the post-advisory exposure time.
At a glance these graphs paint a positive picture. Each ecosystem appears healthy, with the majority of vulnerabilities disclosed in an advisory being remediated very quickly. This demonstrates that security is a priority for most maintainers.
But it should be noted that vulnerabilities where coordinated disclosure was successful will have zero post-advisory exposure time (or even negative time!). In npm and PyPI almost 60% of the vulnerabilities in our database were remediated before the publication of the corresponding advisory. Cargo has a much lower percentage, around 16%; more on that shortly.
Let’s direct our attention to cases that did not see a coordinated vulnerability disclosure. The following histograms show the post-advisory exposure time, excluding successfully coordinated disclosures.
In all three systems, many vulnerabilities are remediated within 30 days of advisory publication. This includes many zero-day exploits, such as log4shell, that were fixed as quickly as possible, even without the more ideal option of a coordinated vulnerability disclosure.
In the case of Cargo, the number resolved in the first 30 days is a staggering 70% of all vulnerabilities remediated after advisory publication. This is because many maintainers choose to release the remediation on the same day the advisory is published, resulting in non-zero but very brief post-advisory exposure time.
The long tail of vulnerabilities with significant post-advisory exposure time is a valuable signal on the health of the corresponding packages. For developers taking on new dependencies, knowing that they will not be left exposed for long periods of time is critical to their security posture. For existing users of a dependency, being aware of changes to future remediation likelihood of potential vulnerabilities is equally important.
Currently it is hard to know how a given package has previously performed according to this metric. Ideally this information would be easily accessible, allowing potential and existing users to make informed decisions about their dependencies.
Mean time to remediation
The number of known vulnerabilities that a package maintainer has remediated in the past can be used to help build trust between maintainers and users of OSS. Additionally, the length of time users of a package were left exposed to known, unremedied vulnerabilities in the past can provide a more detailed characterization of a package maintainer’s response.
In addition to these signals, Mean Time to Remediation (MTTR) has been proposed as a useful indicator of the quality of a package’s maintenance.
However, the available data about advisories rarely contains timestamps for critical events in the remediation process. For example, most advisory databases, including GitHub Advisories and OSV, do not provide a timestamp field for the private disclosure of the vulnerability or the maintainers acknowledgement. And while some advisory write-ups do include an event timeline, these are quite rare.
These missing timestamps make it impossible to compute the time that elapsed between a maintainer being notified of a vulnerability and the release of a remediation, relegating MTTR to a, for now, still hypothetical metric to compute.
Vulnerabilities are an inevitable part of software development. The code reuse and efficiency gains made possible by OSS broadens the potential impact of vulnerabilities.
But cooperation between parties that discover vulnerabilities and package maintainers reduces the time that users are left exposed to publicly known vulnerabilities. Thanks to the hard work of OSS maintainers, there is no post-advisory exposure for the majority of vulnerabilities in our advisory database.
Developers should still prepare for less ideal outcomes. Every dependency they introduce increases the risk of exposure to future vulnerabilities. The clearance rate and post-advisory exposure time for past advisories can provide users of OSS assurance about the quality of maintenance their dependencies receive. While past performance may not always predict future behavior, it can be used as a valuable signal to help make informed decisions.