Vulnerability remediation is the process of finding and fixing security vulnerabilities in your systems—and it’s an especially important discipline in the world of software supply chain security.
Due to the complexities of large organizational systems and lack of available resources, most codebases have known, exploitable vulnerabilities in their runtime applications. This leads to frustration and loss of morale among the people trying to keep codebases safe: board members, C-level leadership, and DevSecOps teams.
It won’t be this way forever. Governments are beginning to increase regulations surrounding cyber security, which will lead to advancements in the tools and skills needed to keep applications secure. Ten years from now, we’ll likely be shocked that applications were allowed to host the degree of vulnerabilities they currently do—similarly to how we find working conditions from past decades and centuries appalling by today’s standards.
However, even if perfect remediation is out of reach today, we can still find ways to do better. This guide discusses how vulnerability remediation works, why organizations struggle with it, the differences between remediation and mitigation, and best practices for setting a strong remediation policy. (Plus, we share how MergeBase’s Dynamic Application Surveillance and Hardening capabilities can further secure your application even while vulnerabilities exist in your codebase.)
This is a long piece, so here are some quick links to the various topics we cover:
I. How vulnerability remediation works
II. Why most companies never remediate all vulnerabilities
III. Vulnerability remediation vs. vulnerability mitigation
IV. Best practices for setting up your vulnerability remediation policy
How vulnerability remediation works
At face value, vulnerability remediation is a straightforward four-step process:
-
Detect vulnerabilities. Your DevSecOps team (with the help of your security tools) finds known exploitable vulnerabilities in your systems, including transferable vulnerabilities from third-party code.
-
Triage remediations. You can’t fix all your problems all at once—and you rarely have to. Some vulnerabilities pose a much greater threat than others, and the time it takes to find and implement a patch can vary from vulnerability to vulnerability. Once you know what vulnerabilities exist in your systems, you need to prioritize the order in which you’ll address them.
-
Patch vulnerabilities. This is the part of the remediation process when you actually apply the remedy. Your DevSecOps team implements patches and updates, so your systems are no longer exposed to the identified threats.
-
Monitor and repeat. When done right, vulnerability remediation is a continuous process. Your DevSecOps team should consistently monitor your systems so they’re ready to repeat the process when new vulnerabilities arise.
This process is initiated, executed, and overseen by people at various levels within your organization equipped by various technologies.
The parties responsible for vulnerability remediation within an organization
While most of the vulnerability remediation process is executed by the DevSecOps team, it involves individuals at several levels of a company’s org chart, from the front-line developers to the board of directors.
The board of directors (risk and/or audit committees)
The driving voice for vulnerability remediation within a company often comes from the board of directors.
Cyber crime is on the rise—in fact, it’s on track to becoming the leading cause of criminal damages in the world. Board members on the risk and audit committees want to make sure the companies they oversee are doing what they should to keep their products, data, and customers safe from bad actors.
The board is responsible for setting their organizations’ risk appetites and keeping them within those parameters. In order to determine whether or not their organizations are operating with an appropriate level of risk, these board members set service-level objectives—the KPIs for security. Board members are also the primary advocates for implementing processes, policies, plans, and tools that enable their organizations to achieve these standards.
While the board usually advocates for the organization to implement the whole four-step vulnerability remediation process, they most directly contribute to Step 2: Triage. This is because the board usually sets expectations for how long vulnerabilities of different threat levels should be tolerated within the application before being remediated.
Board members have long been attentive to network and operating system security. But with the rising threat of third-party vulnerability exploitation and increasing regulatory standards around software supply chain security, board members on risk and audit committees are increasingly seeking ways to make their companies’ software applications secure too.
C-level officers in technology, IT, and security
When board members set expectations for an organization’s software security, it falls to the technical C-level officers to see these expectations met. While it varies from company to company, this usually falls to the individual in the CIO, CTO, or chief information security officer (CISO) role.
The C-level officer functions as a go-between for both the board member and the DevSecOps team: they set up processes and norms for the DevSecOps team on behalf of the board, and they report to the board by summarizing the findings, activities, and results from DevSecOps.
This role is crucial for implementing and improving vulnerability remediation. While remediating vulnerabilities can feel like a pain (or just “one more thing”) for many C-level officers, a leader who takes ownership in this area and drives the team to find and implement stronger security practices can make a significant impact.
The development, security, and operations (DevSecOps) team
These are the practitioners when it comes to remediating vulnerabilities. These are the people who are actively working through the four steps of vulnerability remediation:
- DevSecOps uses specialized tools to detect new known vulnerabilities within the organization’s applications.
- DevSecOps then follows the prioritization policy set by the board to determine the order in which vulnerabilities are fixed.
- DevSecOps may implement patches themselves, or they may push patch requests to other development teams. (It isn’t uncommon for most organizations to implement patches using a mix of DevSecOps team members and practitioners from other teams.)
- DevSecOps reports security status and activities to the organization’s security leadership.
Of course, the DevSecOps team can’t do all this manually. Most organizations use a diverse set of tools to help them by automating elements of the process.
Commonly used vulnerability remediation software tools
DevSecOps teams often use a mix of the following solutions to make the vulnerability remediation process more accurate and manageable.
- Software composition analysis (SCA) tools. This technology scans the third-party code you’ve incorporated in your application for known vulnerabilities. A strong SCA tool (like MergeBase) will not only flag vulnerabilities but also give your developers guidance on fixing them.
- Static application security testing (SAST) tools. SAST tools scan your proprietary code for potential security weaknesses. These tools are often comprehensive but slow.
- Dynamic application security testing (DAST) tools. DAST tools crawl your running application looking for ways that it might be broken or exploited by malicious actors. DAST tools are often used by security assessors and penetration testers to see how a hacker might exploit the target software application.
- Web application firewalls (WAF). A mature solution, firewalls filter and monitor traffic between your solution and the Internet—sometimes blocking traffic altogether. WAFs are a standard buffer against common types of attacks, but they’re not the most effective tools when it comes to recognizing and responding to clever new threats.
- Runtime application self-protection (RASP) tools. A more recent addition to the application security world, RASP tools empower the application to protect itself. A RASP tool monitors activity within the application as it’s running and takes proactive action when it recognizes potentially dangerous behavior. This might include terminating a user’s session, sending a notification to your DevSecOps team, and/or shutting the application down.
Ideally, the process, the people, and the tools would work in harmony to find and fix vulnerabilities as they arise within your applications.
But we don’t live in an ideal world.
Why most companies never remediate all vulnerabilities
Full vulnerability remediation would result in your application being 100% vulnerability free—and staying there. But most companies don’t come anywhere close to this point.
In fact, 2023 Synopsis report on more than 1,700 codebases found that 84% of codebases contained at least one known vulnerability, and 48% contained at least one high-risk vulnerability. And these aren’t “oopsie” gaffs—these are known, documented, exploitable vulnerabilities that are allowed to persist within these companies’ codebases.
This is due to a variety of reasons:
- Shortage of human resources. Sometimes there aren’t enough people to keep vulnerabilities under control—and in some cases, vulnerabilities can crop up faster than people can stomp them out.
- Lack of clear priorities. If a company’s service level objectives (SLOs) aren’t clear and actionable, a DevSecOps team may be caught in approval or triage limbo—they won’t know which vulnerabilities to prioritize.
- Lack of developer guidance. The problem may be known, but finding the solution can take hours or days of research, which can often feel like a lower-priority task than the urgencies of everyday work life. And even if a patch is found, a developer may need to spend more time making sure that implementing it won’t break your application.
- Vulnerability fatigue. If a tool yields too many false positives, a DevSecOps team will often feel less urgency to fix issues as they arise—they don’t want to spend hours hunting down a problem that isn’t there.
- Lack of solutions altogether. Most applications are built using open source libraries, and open source authors are often under no obligation to update their code. This means that an organization’s software may be built on libraries with vulnerabilities for which no patch exists.
- Commercial consequences. If an application needs to be shut down in order to fix a vulnerability, the fallout could be even worse than the vulnerability itself. Some institutions can’t afford to take their systems offline for a few hours, let alone a week.
Because full vulnerability remediation can rarely be achieved, companies often turn to another discipline to secure their applications: vulnerability mitigation.
Vulnerability remediation vs. vulnerability mitigation
While vulnerability remediation eliminates vulnerabilities, vulnerability mitigation allows vulnerabilities to persist within an application and takes steps to reduce the harm that can result from them. It’s a Band-Aid solution: a stop gap for when an organization lacks the time or resources to actually fix the problem.
Vulnerability mitigation is never the preferred way forward—in an ideal scenario, every organization would be able to actually fix every vulnerability that arises within their systems. However, due to the aforementioned hindrances organizations face, not every vulnerability can be remediated. This means that for most organizations, simply mitigating vulnerabilities is an imperfect solution, but it’s better than nothing.
While mitigation may sound like a bad idea, an analogy might be helpful for understanding why some companies choose to use this method in their security practices.
Vulnerability remediation and vulnerability mitigation: an analogy
Let’s assume you run a vending machine company. For decades, you’ve operated thousands of vending machines at gas stations, hospitals, universities, and airports. The machines are old, but functional. Customers put money in the machine, punch in a sequence of buttons, and the machine delivers a snack or beverage.
Now let’s assume that someone discovers that for a certain model of old vending machines, someone can punch in two keys (say, “E” and “8”), and, if they kick the machine in just the right spot, the snacks in slot E-8 will tumble out of the machine free of charge.
Then they post a video to TikTok revealing the hack—and it goes viral. Everybody knows they can use “the E-8 Trick” to score free inventory from vending machines. It’s a known vulnerability.
Now you have a choice to make. You can replace these vending machines with a newer model, but that would be prohibitively expensive. You can send technicians out to fix every single machine that has this vulnerability, but that would be both expensive and time-consuming, especially since these machines are so old that it’s difficult to find technicians who understand them and expensive to find new parts that work with the model. You could take every faulty vending machine offline … but then you’ve killed your revenue stream. Each of these options would be a form of vulnerability remediation—but the solution is more costly than the original problem!
But then you come up with a handy solution: place a sticker on the vending machine saying:
“DO NOT TILT, STRIKE, OR ROCK VENDING MACHINE.
IMPROPER USE MAY RESULT IN INJURY OR DEATH.”
You haven’t solved the problem. The vulnerability persists. But you’ve (hopefully) made it so that a smaller percentage of the population is willing to exploit that vulnerability. Doing this is a form of vulnerability mitigation: you’ve simply reduced the probable harm that this vulnerability could bring about.
This is why vulnerability mitigation is still an option that many companies use today: when the options for remediating vulnerabilities create more problems than they solve, mitigation provides a better-than-nothing means of reducing the damages those vulnerabilities can bring.
MergeBase gives a third option: Runtime Protection
Because the problem of non-remediated vulnerabilities is so rampant in the business world, we built MergeBase with a Runtime Protection option. This feature scans your live application for vulnerabilities, and when a vulnerability is detected, MergeBase simply blocks people from taking the actions necessary to exploit the vulnerability.
To bring it back to our vending machine example, Runtime Protection would be the equivalent of having a tool that automatically scanned the internet for any vending machine hacks. When “the E-8 Trick” is posted online, the tool remotely disables the “8” button immediately after anyone pressed “E.” So if someone punched “E”-”8” sequence, the machine would simply ignore the entry, give an “ERROR” message, and prompt the user to begin the sequence again.
This saves you the trouble and cost of fixing/replacing all your machines, and it prevents people from stealing the inventory in the slot. (You wouldn’t be able to sell from that rack until you fix it, though, so you’re going to face a loss of revenue, but at least your inventory is safe.)
While full remediation is still the goal, Runtime Protection buys you the time you need to deal with vulnerabilities in a resource-friendly manner.
If you want to see how MergeBase Dynamic Application Surveillance and Hardening can bolster your software security, book a demo, and we’ll show you how it works!
Best practices for setting up your vulnerability remediation policy
So if vulnerability remediation is the goal, what’s the best way to make it happen?
Fully remediating every vulnerability might not be a feasible solution for your organization today, but you can still take steps to make your systems more secure. Setting (and abiding by) a vulnerability remediation policy is a foundational approach to maintaining a secure application. If you and your organization want to grow in this area, here are some best practices to follow when setting up your vulnerability remediation policy.
1. Make vulnerability remediation a priority
Stakeholders at every level (board, C-level officer, and DevSecOps) should understand how vulnerability remediation folds up into your organization’s objectives. Remediation can be a painstaking discipline, so it’s crucial for you to communicate how important it is to the teams who are expected to execute on this.
2. Set measurable service-level objectives
Statistically speaking, maintaining a vulnerability-free application (or series of applications) is highly unlikely. Therefore, you’ll need to set realistic and measurable objectives for your team to strive toward.
This includes setting timeframes for remediating vulnerabilities based on threat level. You want high-risk threats to take priority over low-risk threats. You also don’t want low-risk threats to linger in your system for months … or years.
A good way around this is to codify an expected timeframe for dealing with threats at various levels. E.g., a 9.8/10 threat should get your team’s full attention until it’s fixed, but a 1.6/10 threat shouldn’t be allowed to persist in your system for more than a quarter.
3. Align accountability and authority in terms of fixing vulnerabilities
It’s common for a DevSecOps worker to be tasked with asking a leader in a different department to fix an issue—and this can cause problems.
If you’re holding your DevSecOps teams accountable for fixing problems that only another department can fix, your incentive structure is out of alignment. Ideally, the onus of remediating vulnerabilities should fall to the people who have the power to fix those vulnerabilities.
When you create your vulnerability remediation policy, make it very clear that the people held accountable for remediation have the authority to remediate vulnerabilities.
4. Continuously remediate—don’t do it in batches
It might be tempting to do a scan of your software every month or quarter, but today’s cyber crime environment demands continuous scanning and monitoring of your systems for vulnerabilities. You need to know exactly where and when you’re vulnerable to bad actors.
The best way to do this is by continuously working through the four-step vulnerability remediation process. You should be constantly scanning your application for vulnerabilities, prioritizing efforts, patching, and monitoring for future issues.
5. Give your team the resources and tools they need to uphold your remediation policy
Scanning and prioritizing vulnerabilities is not a task you want to handle manually. It’s important to invest in tools that can automate as much of this process as possible—because it isn’t realistic to expect any human (or team of humans) to catch and fix every vulnerability on their own.
One way to set your DevSecOps team up for success is by getting them a vulnerability scanning tool that provides your developers with guidance regarding:
- The threat level of a given vulnerability
- What patches are available
- How compatible those patches are with your current applications
- How to implement those patches
In your best case scenario, the tool automatically implements compatible patches that won’t break your application.
Furthermore, it’s important to invest in a tool with a low false positives rate. False positives (reports of vulnerabilities that don’t actually exist) are a wild goose chase for your developers, and our research has found that they’re harmful to both DevSecOps productivity and inter-team morale.
Get better, more practical protection with MergeBase
Organizations will continue to struggle with the problems of persistent vulnerabilities within legacy code. In the future, we’ll wonder how we ever operated at the levels of risk exposure we do today, but we’re not there yet.
While full remediation is rarely attainable today, there are ways that we can keep our software more secure without settling for Band-Aid mitigation solutions. MergeBase’s Dynamic Application Surveillance and Hardening is one way to keep your application safe and buy your team the time they need to fix the problem. (And MergeBase is the only SCA tool on the market that offers this.)
If you would like to see how MergeBase can protect your applications from known vulnerabilities in runtime, we’d love to show you how it works.
