The systems in place to manage software vulnerabilities are often overwhelming and ineffective. But that can change if teams enlist their end users to remediate vulnerabilities.
Organizations that implement an effective vulnerability management program benefit in two ways. First, they are compromised less often. Second, if a single device does become compromised, the degree of lateral movement an attacker can enjoy is significantly constrained.
But even though endpoint vulnerability management has always been vital to cybersecurity, it’s a tough nut to crack. That’s especially true when you take a broad view on your fleet’s vulnerabilities, which could include everything from shoddy passwords to EOL software. But even if we narrow the scope to patch management for software vulnerabilities, we’re talking about a major challenge.
I recently came across a sobering 2020 study by the Ponemon Institute about on-premise and cloud vulnerabilities. The following statement leapt off the page:
“Fifty-three percent of respondents say their organizations had a data breach in the past two years. […] of these data breaches, 42 percent of respondents say they occurred because a patch was available for a known vulnerability but not applied."
Meanwhile, Verizon’s 2024 Data Breach Incident Report shows that the problem is only getting worse. As they put it, in 2024 “attacks involving the exploitation of vulnerabilities as the critical path to initiate a breach…almost tripled (180% increase) from last year.”
A failure in detection is unfortunate but easy to understand and forgive. As IT administrators and security practitioners, we know it is not possible to enumerate all potential flaws in installed software.
That isn’t the case here. Here, the problems are known in advance and on a list of things to be fixed! What we have isn’t a failure in detection but a failure in our remediation processes. In many organizations, these processes are either totally broken or simply do not scale.
To fix these processes, we need to understand the root causes that left them so flawed. Luckily, we don’t have to look far as many of these problems are rooted in the fundamental design of the vulnerability program itself.
The naive top-down approach
When organizations plan their first vulnerability detection and management programs, they tend to approach it by following these steps:
Create a central inventory of the software across your fleet. (1Password® Extended Access Management does this asset management for you)
Parse vulnerability data feeds like the National Vulnerability Database, usually with some form of vulnerability scanning software.
Compare the data between the feed and the inventory to find your list of software vulnerabilities.
Filter and prioritize vulnerabilities based on the CVSS score or other criteria.
Create an alerting/notification capability to drive vulnerability remediation.
I call this a top-down approach because it starts from a list of all known software vulnerabilities and then works downward, applying them to devices in an organization’s fleet.
On the surface, this appears to be an approach that should reduce the risk of serious compromise. But does it help in practice?
The answer is no.
Classic vulnerability management does not yield actionable remediation
The top-down strategy is designed to send the maximum amount of information to a small team of expert human operators. Then, they’re expected to triage it and drive remediation.
Unfortunately, this approach only works well if the following is true:
An operator can quickly review all possible use cases of any device.
All aspects of any device – from the operating system, to the software, to the configuration – can be managed remotely and in a scalable way.
There is always a safe way to remediate the vulnerability (or mitigate the risk).
For most real life situations, that’s a fairy tale.
In the real world, it’s pretty standard for vulnerability management software to enumerate long-flashing red lists of software vulnerabilities that:
Have no vendor supported remediation path.
Relate to software that is not important or even used.
Overstate their applicability and their severity.
Will be fixed automatically without any intervention.
Are just straight-up false positives.
While these problems affect both servers and endpoints, it is the end-user-driven endpoints that are a terrible fit. IT and security teams are inundated with long lists of software vulnerabilities, most of which will be low quality and non-actionable. Soon, alert fatigue will set in, and the entire program is put at risk.
To compensate, IT teams often do precisely the wrong thing. Rather than change their vulnerability management process, they force the endpoints to change to best suit it. For end-user-driven devices, this means maximizing the amount of remote management IT does on the device, and minimizing the management tasks typically done by an end-user.
But end-user devices are multi-purpose, and those purposes cannot be fully known in advance. It’s nearly impossible for a skilled operator to triage vulnerabilities correctly. This creates a negative feedback loop where the operator will seek more data and more control over these endpoints.
Counterintuitively, this just makes the problem worse. The data needed by the IT team to make good decisions on behalf of the user starts to feel like dystopian-level surveillance. End-users begin to notice the performance and productivity impacts of endpoint management software. It’s not too long before an employee either works around it or starts using personal devices instead. When this ultimately happens, instead of reducing risk, you’ve eliminated your ability to enumerate it.
An end-user focused approach to vulnerability management
In security, we often let perfect be the enemy of good. This makes us blind to novel solutions that can significantly reduce (but not eliminate) the likelihood of a device becoming compromised.
One of those novel solutions is to enlist the help of the end-users directly and have them remediate serious vulnerabilities for us. As we wrote in Honest Security:
End-users can make rational and informed decisions about security risks when educated and honestly motivated.
To apply the values of Honest Security to vulnerability remediation, we need to devise a system optimized for end-users. This means:
Encouraging preventative behaviors (like keeping the OS and apps up-to-date) rather than reacting to individual vulnerabilities.
Ensuring issues reported to end-users are always accurate, instead of an exhaustive but potentially flawed list of vulnerable software.
Sacrificing reporting speed in favor of waiting for vendor-supported patches to be available..
Focusing only on the high impact vulnerabilities.
This approach does not give you “perfect” vulnerability management. That’s ok because we aren’t looking for perfection. Instead, we get an imperfect, yet more effective tool to manage the nuances of patching vulnerabilities, particularly in apps where traditional automation cannot work. That is a significant upgrade from classic top-down vulnerability management.
Apply these principles to device posture Checks
In the 1Password Extended Access Management device trust product, our Checks feature allows IT administrators to verify that their devices are meeting specific requirements in their security posture. For instance, our device trust solution ships with a check to ensure Google Chrome is always up-to-date.
In addition, it also deploys checks to look for known violations of IT policy or high-impact security risks. This can include:
Evil applications (e.g. malicious browser extensions)
Misappropriated sensitive data or credentials (e.g. Incorrectly stored 2FA backup codes)
Specific high-impact software vulnerabilities
When a highly publicized new vulnerability hits the news, customers will frequently ask us if we will build a check that detects it. With our end-user focus, we approach these decisions by answering the following questions:
Does this vulnerability easily allow for remote code execution with minimal or no user interaction?
Is it unlikely the vulnerable software will update itself within a reasonable time frame without end-user or IT intervention?
Is there a straightforward vendor-provided patch available?
Is the likelihood low that this software will have a similar issue in the next 90 days?
If the answers to all of the above are a resounding yes, then it meets the criteria for end-user remediation, and therefore, it makes sense for us to ship a new check.
To demonstrate why these questions matter, let’s work through an example where we created such a check. Specifically, let’s talk about CVE-2019-9535, a 2019 vulnerability in popular terminal software iTerm2. To summarize, this vulnerability allows an attacker to craft specific output that, when viewed in the terminal, could cause the software to execute arbitrary commands. Now that we understand the nature of the vulnerability let’s walk through the checklist above.
Does this vulnerability easily allow for remote code execution with minimal or no user interaction?
Yes. According to the report, during the ordinary course of using the app, a user can be easily tricked into running the exploit. It can then be leveraged to compromise the integrity of their device completely.
If this vulnerability didn’t result in an RCE, and simply crashed a device, it wouldn’t be worth fatiguing the user with notifications.
Is the likelihood low that the vulnerable software will update itself within a reasonable time frame without end-user or IT intervention?
Yes. iTerm2 is not an auto-updating app (more on this later). While it does prompt users when an update is available, these updates can be easily skipped by users. This means we cannot even be reasonably sure that all the vulnerable versions running in our environment will get patched.
If this app auto-updated, and the likelihood of the vulnerability being exploited within 30 days was low, it wouldn’t be worth fatiguing the user with a notification.
Is there a clear vendor-provided mitigation available?
Yes. Simply following the update prompts will immediately patch the vulnerability. An end-user will have no trouble mitigating the risk, and we should have no problem writing step-by-step instructions that anyone familiar with the app can follow.
Why does this matter? It’s common for severe vulnerabilities to be reported days or even weeks before a vendor can develop a mitigation or a patch. Sometimes the vendor-supported solutions never arrive. We must wait for the most user-friendly solution to become available so that we can drive the maximum impact through end user notifications. Fatiguing end users with issues they can’t fix will simply train them to ignore the ones they can.
Is the likelihood low that this software will have a similar issue in the next 90 days?
Yes. We can’t predict the future, but iTerm2 historically has’t had many vulnerabilities, and it is unlikely to have another with a similar severity in the next three months.
This is important; we don’t want to fatigue end-users with a deluge of issues for the same application (especially when they just fixed it). If we think another vulnerability will be reported soon, we may either want to wait a few weeks to see how the situation develops.rIf the app is essential, we can add it to a list of apps we always ask users to keep up-to-date.
As you can see from the above example, we only want to contact end users about serious problems, and only when we can provide a clear path to remediation.
As for what that path is, we’ll use 1Password Device Trust’s method as an example. Once we’ve determined that we want end users to apply a specific patch, we’ll create a Check that uses osquery to see which devices have an unpatched version of the software.
For any device that does, 1Password Device Trust will send a menu bar alert with clear instructions on how to apply the patch. It will also provide consequences. If a user fails to remediate vulnerabilities within a certain amount of time, then they won’t be able to authenticate into company systems. That way, your team has peace of mind that device vulnerabilities can’t be used to breach systems. While the above rubric can be applied to most apps, auto-updating apps, like web browsers, change the calculus significantly. Let’s discuss.
What about evergreen apps?
Many of our most important apps manage their own update process outside of the operating system’s central software update. Apps that do this are often known as evergreen apps, and they can significantly complicate the vulnerability management remediation story.
Plenty of apps automatically keep themselves up-to-date without the user needing to participate at all (not even by clicking an “install update” button). In theory, every time you launch the app, you are using the latest version. Modern web browsers are the canonical example, with Google popularizing the practice shortly after its Chrome Browser was released in 2008.
Auto-updating apps have significant advantages over apps with different release and update strategies. Before the majority of the web browser industry became evergreen, developers would often be stuck supporting web browsers that were years old. Today, if a vendor ships a new release, web developers can expect that release to become the dominant version in less than a month.
As I write this blog post, I am currently running the following auto-updating apps: Google Chrome, Slack, Discord, and VSCode. To be honest, I’m not even sure if that is all of them. The benefit of the auto-update process is how seamless it is; as a user, I simply don’t think about it.
How to reason about the vulnerability of auto-updating apps
Remember the old philosophical question: “If a tree falls in the woods, but no one is around, does it make a sound?” Well, with auto-updating apps, we have a similar one to consider – if an app is vulnerable on disk but when launched is immediately updated to the non-vulnerable version, was it even vulnerable to begin with?
The answer – as I’ve learned from experience – is “it depends.” Let’s explore what important information we need to know the answer definitively.
We need to understand the probability of an exploit successfully running against a vulnerable auto-updating app before it has a chance to upgrade itself.
To do that, we need more data about the app itself, the device that’s running it, and even the person who uses that device. We can split this into two simple criteria.
Criteria #1: Is it easy to activate the exploit?
For example, let’s take an attacker attempting to leverage an RCE-style vulnerability. This attacker will likely take a two-step approach: build a reliable exploit and then find a reliable way to get their victims to trigger it. Knowing how likely a victim is to trigger the exploit is vital information, and is different for each device and end-user you are considering.
To enumerate this risk, we often think about the following (ordered by severity):
Can the attacker trigger the exploit via a communication channel that pushes untrusted content to end-users? (e.g. email, SMS, iMessage, etc.)
Can the attacker craft a malicious URL that causes the app to launch and immediately load the exploit?
Can the attacker craft a malicious file that causes the app to immediately load the exploit when opened?
Is the exploit so ubiquitously distributed that a user is likely to encounter the malicious payload through the normal course of their duties?
If we can answer in the affirmative to at least two of the three criteria above, then yes, this exploit is a threat.
Criteria #2: After launch, how quickly does the app update?
When an update is released, evergreen apps are supposed to apply the update quickly. Unfortunately, this isn’t always the case. Not all apps poll for updates as frequently as you’d expect, and if the update process isn’t working correctly, an important update may never be applied. An obvious sign is when you see a very old version of an auto-updating app running.
The real devils can be found in the weird nuances around how an app auto-updates. More specifically: after the update, can multiple versions of the app exist simultaneously?
There are two main auto-update strategies, each with slightly different risks:
Seamless full update
This is the best strategy. The update is downloaded and applied to in the background, and the app restarts without any user interaction.
User action required
This is the worst strategy. Even after the update is downloaded and applied to the app in the background, the old code still runs until the user takes action (like entirely restarting the app).
Three approaches to autoupdate vulnerability management
You want to pick one of the following vulnerability management strategies for each app based on the criteria above.
Strategy #1: Ensure app is always up-to-date
In this strategy, we inform the user every time the app is out of date, within a grace period. A good example is the Chrome Check in 1Password Device Trust.
This strategy is a good fit for common, regularly used apps that receive a continuous stream of security updates. 1Password Device Trust chooses this for web browsers and other apps with regular exposure to untrusted user-generated content (webpages).
In this strategy, the key to minimizing fatigue is only to generate an issue if we detect the app is in-use and out of date, unless there is a pathway to activate the exploit through opening an arbitrary file or via a link.
Strategy #2: Ensure app has the latest security-related update applied
This strategy is similar to #1, but instead of requiring all updates to be applied, we need to understand the release announcement and its security impact before asking a user to perform the upgrade.
Strategy #3: Ensure app auto-update system is working
This strategy detects if a regularly launched app is significantly out of date. This is a good indicator that the app’s automatic update is damaged or has been disabled intentionally by an advanced user.
This is a good strategy for apps with similar exposure, but fewer vulnerabilities to a web browser. Slack is an excellent example of this. It’s an app that really should always be up-to-date. If it isn’t, it’s a great indicator that something has gone wrong.
Pulling it all together
As we’ve learned above, we need to let go of any preconceived notions of what makes an effective vulnerability management program. That will open our minds to novel ideas that allow us to make real headway on securing user devices.
The key is to distill down the use cases that end users can solve on their own, and use tools that allow you to effectively communicate with them at scale, enabling them to keep their devices secure. This has the added benefit of freeing up time and toil for admins who were previously inundated with an ever-growing list of CVEs – many of which are totally unimportant– to manage for their entire fleet.
If the issues we ask our users to fix don’t feel important, are wrong, or aren’t actionable, we will lose their trust. When that happens, we’ve lost one of the most effective resources we have to solve our most challenging and nuanced security issues.
If you are interested in learning more about end-user-driven remediation and the philosophy behind it, I encourage you to read Honest Security. If you want to get started with a turn-key product built on top of these principles, you should check out 1Password Extended Access Management.
Jason Meller
VP, Product Management
Tweet about this post