Blog

Category: Software Supply Chain Security

Learn about Software Supply Chain Security risks, standards, and more!

Suppression Management: How to Balance Software Supply Chain Security with Practical Business Goals

In an ideal world, you would only ever ship immaculately secure versions of your software—free from all known vulnerabilities. 

However, you’re not building software in an ideal world: the security of your product is just one of many factors that affect what you ship and when you ship it. Budget constraints, talent gaps, operational schedules, and good old-fashioned customer deadlines also come into play—which means software companies often need to sacrifice short-term cybersecurity perfection for the sake of long-term product (and business) viability.

This happens in every area of life, not just software. We prop the front door ajar while moving heavy boxes in or out of the home. We trust bartenders to hold on to our credit cards when we start tabs. We trust restaurant cooks to handle our food in a sanitary manner, even though we don’t directly observe them washing their hands. All of these decisions sacrifice some security for the sake of practicality—because if we were 100% secure all the time, we’d never get anything done.

When it comes to the security of your software products, you don’t want to be held hostage by your own security policies. You need to be able to temporarily (and prudently) loosen your security standards in order to keep the business running. You also need to be able to follow up and patch those vulnerabilities later.

That’s why MergeBase now offers suppression management capabilities. MergeBase lets you strategically suppress your security tools so you can ship your products while tracking the vulnerabilities you allow to pass through so you can fix them later.

Suppression management with MergeBase

There will likely come a time when you discover a new vulnerability in your software supply chain, but you don’t have the capacity to address it before your shipping deadline. An advanced SCA (like MergeBase) will prevent you from shipping a software build that contains high-risk vulnerabilities—but what if you need to meet your deadline today and can easily fix this problem in the next 72 hours?

When this happens, MergeBase allows you to suppress the vulnerability and ship your product. You can use our suppression management feature to choose which vulnerabilities to ignore and for how long to ignore them. 

This allows you to stay on schedule and quickly patch the problem afterward.

This exception to the rules is recorded, so you have full accountability when it comes to the issue. MergeBase logs the vulnerability, the time period for which you’ve decided to ignore it, and any rationale you choose to provide as comments. You’re still upholding your security policies—you’re simply making (and recording) a judgment call.

Importantly: this is a controlled exception. You’re not indefinitely disabling all of your vulnerability protection. Using this feature allows you to make a conscious, temporary exception to the rules. 

It’s similar to how you might respond if a high-ranking sales employee showed up at the office one morning and, upon attempting to scan in, realized she had forgotten her company ID at home. You wouldn’t call the police, you wouldn’t fire her, and you likely wouldn’t send her back home to get her badge. But you also wouldn’t disable all the locks in the building for the day and risk allowing just anyone inside.

Instead, you’d give her a temporary visitor pass, remind her of the importance of bringing company ID to work with her, and let her do her job that day. If the problem persisted, you’d take further action. 

Likewise, suppression management allows you to temporarily suspend the status quo—without upending your entire software supply chain security policy.

How suppression management works in MergeBase

The first step in suppression management in MergeBase is to set your baseline policies for what kinds of vulnerabilities you permit to exist in your software. MergeBase detects the threat level of every known vulnerability on a scale of 0–10, which means you can preconfigure MergeBase to halt any build from going live if it contains vulnerabilities above a certain threat level.

You can also permanently suppress common vulnerabilities and exposures (CVEs) in MergeBase. This is useful if your product or operations have built-in security measures that will naturally prevent a given CVE in your software supply chain from harming your customers or their end users. (Just like how one might ask their guests to take off their shoes upon entering the house, rather than only unlocking the front door after a visitor provides proof that they’ve already removed their shoes.)

It’s important to note that suppression management in MergeBase works at the individual vulnerability level—not the software component level. MergeBase will never give a component a unilateral free pass: you need to manually specify which vulnerabilities get temporary or permanent suppression.

Improve your security and productivity with MergeBase

As software supply chain security law matures, software companies like yours need to manage their abilities to stay both secure and productive. MergeBase can help you do both.

If you want to explore how MergeBase can help you strike the right balance, book a demo with us—we’d love to learn more about you and show you how we can help!

Book a Meeting

Industry Report: The True Costs of False Positives in Software Security

False positives: A wild goose chase (Photo: Chandler Cruttenden)

You’ve been told that there’s a vulnerability in your software application. You’ve spent hours trying to find the problem. You’ve double-checked the alert. You’ve double-checked the software. No patches are available. 

Finally, after ruling out every other option, you’re forced to conclude that there was never a vulnerability in the first place. You’ve just wasted your whole afternoon trying to fix a problem that wasn’t there. 

This is the problem with false positives in software security.

False positives are false alarms—and each one can send you on a wild goose chase that costs your organization time, money, and emotional overhead. 

That’s why it’s important to invest in security tools that reduce false positives. The more accurate your security software, the more easily you can mitigate the very real costs that false positives represent to your business.

We wanted to understand how false positives affect security teams, so in August 2022, we surveyed cyber security professionals on the subject. Twenty-nine individuals shared their thoughts on how false positives impact organizations.

Survey results: How software security professionals feel about false positives

We surveyed software security professionals on the costs of false positives, and the results indicate that false alarms are not only economically costly but also have long-term negative effects on the teams that deal with them.

In fact, given their businesses’ resources and constraints, most of our respondents would rather reduce false positives than increase true positives. False positives are a big problem in software security today.

Survey-Results-Metric-to-improve

This report will dig into these findings one by one, but here’s a summary of what our survey found:

  • 72% of respondents believe that false positives damage team productivity.
  • 62% of respondents would rather immediately reduce false positives than immediately catch more true positives. 
  • 62% of respondents also believe false positives damage overall team morale.
  • 59% of respondents claim that, on average, false positives take more time to resolve than true positives.
  • 55% of respondents believe false positives damage relationships between teams in their organization.

False positives, true positives, and the confusion matrix

A false positive occurs when your tool detects a vulnerability that isn’t really present. If you’re not familiar with statistical classification, it might be helpful to take a moment to get an idea of what we mean by “false positives.” (If you already know the difference between a false positive and a true positive, feel free to skip this section.)

You monitor the security of your software using various scanning tools, which check for vulnerabilities in your product. These tools might include static application security testing (SAST), dynamic application security testing (DAST), and/or software composition analysis (SCA). If a tool doesn’t find any vulnerabilities, it reports a negative—but if a tool detects a vulnerability, it reports a positive.

What the tool says

However, that’s just what the tool says. There’s the tool report, and there’s the actual truth: there either is a vulnerability (positive), or there isn’t (a negative).

there’s the actual truth

In an ideal world, our tools would always match reality, but tools can make mistakes. If the tool report matches reality, then the report is true. If the tool doesn’t match reality, then the report is false.

To illustrate this mismatch, programmers use what’s (appropriately) called a confusion matrix. This compares the tool reports to the actual condition of your software, and it looks like this:

The Confusion Matrix in Software Security

So there are two kinds of positives your software security tools are going to give you: true positives and false positives. True positives alert you to real vulnerabilities in your software. A false positive is a false alarm. 

We queried software security professionals on this matter, and the responses indicated a consistent sentiment: false positives waste valuable dev time and damage team morale.

False positives take longer to resolve

When a scanner detects a vulnerability, the security team gets to work fixing it. If it’s a true positive, the issue is resolved when the vulnerability is patched. 

However, if it’s a false positive, the issue can only be resolved when the security team can demonstrate that there’s no real vulnerability present. In other words, a false positive is only resolved once you can prove it’s a false positive.

We asked software security professionals which task, on average, takes more time to resolve: true positives or false positives. Of the people we surveyed, 58% said false positives take more time to resolve than true positives.

false positives take more time to resolve than true positives

It follows that false positives generally damage team productivity. Because resolving false positives detracts from more useful activities (like building your software or fixing true vulnerabilities), false positives can put a good deal of drag on team productivity. 

Our survey responses reflect this: seventy-two percent of respondents agreed with the statement that “False positives damage team productivity”—including twenty-eight percent who said they “strongly agree.”

False Positives damage team productivity

Reducing false positives frees up dev time for things that actually contribute to your business.

False positives damage team morale, leading to vulnerability fatigue

In addition to the economic costs of false positives discussed above, false positives come with several soft, hidden costs. Our survey brought several examples of this to the surface—one of which was the sentiment that false positives erode team morale.

While this sentiment isn’t as pronounced as the sentiments discussed earlier, responses do skew toward agreement with the statement, “False positives damage overall team morale.”

False Positives damage team morale

This erosion of morale can lead to “vulnerability fatigue” or “patching fatigue”—which happens when a team becomes desensitized to vulnerability alerts (whether they’re true or false positives doesn’t matter). An abundance of false positives can weaken even the most zealous team’s motivation to patch vulnerabilities.

And once that attitude works its way into your operations, it can be very, very difficult to reverse.

False positives can damage relationships between teams

A slight majority of respondents agreed that false positives damage relationships between their teams and other teams within their organizations. 

False Positives damage relationships between teams

From an internal political perspective, false positives can easily undermine a security team’s credibility. 

Security teams are already fighting an uphill political battle:

  • Teams dedicated to security usually aren’t building the product. In some organizations, “builders” can see the security team as lower-status contributors, non-contributors, or even a net negative to the business entirely. False positives can reinforce this mindset.
  • Security teams have the thankless task of pointing out where other teams made mistakes. People generally don’t like hearing about how they messed up, and so your other teams have a built-in psychological incentive to discredit your security team. False positives can perpetuate the narrative that security is either not credible or incompetent, which makes it more difficult for them to be taken seriously when a true positive comes along.

Unfortunately, because false positives take more time to resolve than true positives, other teams are more likely to remember false positives. Even if most alerts turn out to be true positives, it doesn’t take many false positives for a security individual or team to get labeled as the proverbial boy who cried wolf in the office.

Drive down false positives with MergeBase SCA

False positives are an unfortunate side effect of vigilant tools, but you can reduce false positives by investing in high-accuracy vulnerability scanners. 

At MergeBase, we’ve built our software composition analysis tool with accuracy in mind. When customers switch to us, they commonly report a significant decrease in false positives and an increase in true positives.

For example, many of our customers are former users of OWASP Dependency-Check, a free SCA solution. Dependency-Check was built to be a super-vigilant vulnerability checker and therefore has a bias toward positives—including false positives. When companies upgrade to MergeBase (a premium SCA), they see an immediate decrease in false positives. 

If you compare MergeBase’s performance against OWASP Dependency-Check’s on a set of libraries pre-seeded with publicly-known vulnerabilities (like this one), the difference in false positives (and true positives) becomes abundantly clear: MergeBase catches more vulnerabilities while reducing false positives.

MergeBase catches more vulnerabilities while reducing false positives.

We’ve built a tool that catches more vulnerabilities and sounds fewer false alarms when it comes to scanning your software supply chain—and if you’d like to see how it can help mitigate the costs of false positives for your team, we’d love to give you a demo!

Book a Demo

Ubuntu Docker Container Scanning – Is it enough to protect your app?

In this post, we are going to look at Docker container scanning:

  1. First, we will build an Ubuntu Docker container, add some software to it, run a Docker scan, and record the vulnerabilities.
  2. Then, we will try to resolve some of those vulnerabilities, run another scan, and examine the results.
  3. Finally, we will determine whether container scanning is enough to protect your applications and data from compromise. 

Building the Docker Container

We will build our Ubuntu Docker container using the Ubuntu Xenial image. Since it is a few versions behind, it will hopefully contain a few vulnerabilities. We are also going to install Python, PIP3, and a vulnerable version of urllib3. If we search the CVE database, we find CVE-2021-28363. According to this CVE, the urllib3 library 1.26.x before 1.26.4 omits SSL certificate validation in some cases involving HTTPS to HTTPS proxies.

The first step in our process is to pull the Ubuntu Xenial Docker image from Docker Hub and run it as a container on our Docker host. The command below will get the image, start the container in interactive mode, and name it ubuntu_xenial. 

docker run -it –name ubuntu_xenial ubuntu:xenial

Once the process completes, you will be presented with the Ubuntu prompt, as shown in the text output below.

PS C:\Users\docker> docker run -it --name ubuntu_xenial ubuntu:xenial

Unable to find image 'ubuntu:xenial' locally

xenial: Pulling from library/ubuntu

80bce60046fa: Pull complete

55a738a15540: Pull complete

e19cf0706c62: Pull complete

de4cdd6c27d1: Pull complete

Digest: sha256:9775877f420d453ef790e6832d77630a49b32a92b7bedf330adf4d8669f6600e

Status: Downloaded newer image for ubuntu:xenial

root@a317929cf5d7:/#

The next step is to install Python and PIP by running the following command:

apt update && apt install python3-pip

root@a317929cf5d7:/# apt update && apt install python3-pip

Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]

Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]

Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]

Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [2051 kB]
.....
Fetched 19.4 MB in 42s (458 kB/s) 

Reading package lists... Done

Building dependency tree

Reading state information... Done

5 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done
.....
The following packages will be upgraded:

libc6

1 upgraded, 85 newly installed, 0 to remove and 4 not upgraded.

Need to get 98.7 MB of archives. 

After this operation, 289 MB of additional disk space will be used.

Do you want to continue? [Y/n]

 

After installing Python and PIP, we are ready to install urllib3. However, since the development team fixed the vulnerability in version 1.26.4, we need to install an earlier vulnerable version. For the purposes of this exercise, let’s install version 1.26.0 from requirements.txt. 

We can check that our requirements.txt file includes our vulnerable version by running the cat command.

root@a317929cf5d7:/# cat requirements.txt

urllib3==1.26.0

We can use PIP to install the  requirements.txt file.

pip3 install -r requirements.txt

root@a317929cf5d7:/# pip3 -r requirements.txt

Collecting urllib3==1.26.0

  Downloading https://files.pythonhosted.org/packages/7e/a7/746338eb8addda2e7662ee5e10a9f85150aba013cd610c9569c17146b914/urllib3-1.26.0-py2.py3-none-any.whl (136kB)

    100% |################################| 143kB 3.9MB/s

Installing collected packages: urllib3

Successfully installed urllib3-1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

root@a317929cf5d7:/urllib3_test# pip3 freeze requirements.txt

urllib3==1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

Now that we have updated our container, it is time to commit it and create our image by running the following Docker command from the docker host.

docker commit ubuntu_xenial

PS C:\Users\docker> docker commit ubuntu_xenial

sha256:130de6a9e3822dd3bbf19131469f3de441967756cd6faf70f52c92183ddea37e

Let’s check that Docker created the image by running the docker images command.

docker images

As we can see in the text output below, Docker created the image. 

PS C:\Users\docker> docker images

REPOSITORY   TAG      IMAGE ID      CREATED              SIZE

<none>      <none>    130de6a9e382   About a minute ago   465MB

ubuntu       xenial   9ff95a467e45   12 days ago          135MB

However, to make our lives easier, let’s give the new image a tag to identify it by running the following command.

docker tag 130de6a9e382 our_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag 130de6a9e382 our_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED        SIZE

our_ubuntu_image  latest   130de6a9e382   3 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago     135MB

As Docker’s scanning feature only works on images and not containers, we are now ready to scan the image for any vulnerabilities. Let’s start the Docker container scanning by running the following command.

docker scan our_ubuntu_image

PS C:\Users\docker> docker scan our_ubuntu_image

Docker Scan relies upon access to Snyk, a third party provider, do you consent to proceed using Snyk? (y/N)docker y

/ Analyzing container dependencies for our_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 193 issues, as shown in the text output below.

Package manager:  deb

Project name:     docker-image|our_ubuntu_image

Docker image:     our_ubuntu_image

Platform:         linux/amd64

Licenses:         enabled

Tested 185 dependencies for known issues, found 193 issues.

Docker Container Scanning Results

If we study the vulnerability report, we find that Docker container scanning highlights some vulnerabilities that can be fixed, as shown in the text output below.

✗  Low severity vulnerability found in glibc/libc-bin

  Description: Improper Data Handling

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-345675

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

✗ Low severity vulnerability found in glibc/libc-bin

  Description: Integer Underflow

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-571388

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

If we look at the message for the two vulnerabilities in the image, we need to update those particular packages to resolve the issue. First, we need to create a container from the new image to make sure we are working with the latest version. As we did previously, we need to execute the docker run command as shown in the image below.

docker run -it –name our_ubuntu_image our_ubuntu_image

PS C:\Users\docker> docker run -it --name our_ubuntu_image our_ubuntu_image

root@0dfcbff2e3d1:/#

After executing the docker run command, we can run the following commands to update Ubuntu from the container’s command line.

apt update && apt upgrade

root@0dfcbff2e3d1:/# apt update && apt upgrade

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease

Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease

Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease

Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease

Reading package lists... Done

Building dependency tree

Reading state information... Done

4 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

The following packages will be upgraded:

  apt libapt-pkg5.0 libc-bin multiarch-support

4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Need to get 2458 kB of archives.

After this operation, 70.7 kB of additional disk space will be used.

Do you want to continue? [Y/n]

After running and installing the updates, we need to commit and create another Docker image to save the changes. 

docker commit our_ubuntu_image

PS C:\Users\docker> docker commit our_ubuntu_image

sha256:bf5809ae94d91ea37ac94fcc7260d607d15925347267bba911e9d3e2308dd93f

We can check that Docker created the image successfully by running the docker images command.

docker images

As we can see in the text output below, Docker confirms the new updated image exists.

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED          SIZE

<none>             <none>    bf5809ae94d9   2 minutes ago    476MB

our_ubuntu_image   latest    130de6a9e382   19 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago      135MB

As we did before, let’s give the new image a tag to identify it by running the following command.

docker tag bf5809ae94d9 updated_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag bf5809ae94d9 updated_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY             TAG       IMAGE ID       CREATED          SIZE

updated_ubuntu_image   latest    bf5809ae94d9   3 minutes ago   476MB

our_ubuntu_image       latest    130de6a9e382   21 minutes ago 465MB

ubuntu                 xenial    9ff95a467e45   12 days ago       135MB

We are now ready to scan the updated container to see if our efforts resolved any vulnerabilities. As before, to run docker scan execute the following command. Note that we are now scanning our updated image.

docker scan updated_ubuntu_image

PS C:\Users\docker> docker scan updated_ubuntu_image

/ Analyzing container dependencies for updated_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 191 issues, as shown in the text output below. This result demonstrates that updating our Ubuntu container resolved two vulnerabilities. 

Package manager:    deb

Project name:      docker-image|updated_ubuntu_image

Docker image:      updated_ubuntu_image

Platform:          linux/amd64

Licenses:         enabled

Tested 185 dependencies for known issues, found 191 issues.

However, if we search both the original vulnerability report and the report generated after our updates, neither report contains an alert for CVE-2021-28363. If you recall, when we created our image, we installed a vulnerable version of urllib3. Since neither scan picked up this vulnerability, we need to consider alternatives that mitigate this risk.

MergeBase Vulnerability Analysis and Resolution

To test the validity of Docker scan’s analysis and reporting, let’s run the same scans using MergeBase.

As in the first example, we first need to run a scan on the Docker image named our_ubuntu_image before we run any Ubuntu updates. 

First, we need to get the docker image repository and tag by running the following command:

docker images

As we can see in the text output below, the repository name is ‘our_ubuntu_image,’ and its tag is ‘latest.’

REPOSITORY TAG IMAGE ID CREATED SIZE

our_ubuntu_image latest 130de6a9e382 18 minutes ago 462MB

ubuntu xenial 9ff95a467e45 3 weeks ago 135MB

Using the MergeBase command-line tool, we can scan this image using the following command:

java -jar mergebase.jar –mode=profile –name=our_ubuntu_image our_ubuntu_image:latest

If we look at the summary scan report on the MergeBase portal, we can see that it found 301 vulnerabilities. This number exceeds the 193 found by Docker scan by 108 vulnerabilities.

Below is the text output from the command-line tool. As you can see, it details multiple CVEs and categorizes them by severity.

java -jar mergebase.jar --mode=profile --name=updated_ubuntu_image updated_ubuntu_image:latest

Saving application profile to https://demo.mergebase.com

  Vulnerable Files Overview

  =========================

  CRITICAL (12)

  -----

  CVE-2017-7614, CVE-2017-7226, CVE-2017-6969 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-8283 (dpkg-dev@1.18.4ubuntu1.7.UBUNTU)

  CVE-2017-8283 (dpkg@1.18.4ubuntu1.7.UBUNTU)

  CVE-2019-18218 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2008-1530 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2016-6309 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2017-12814 (perl-base@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2017-12814 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1126 (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2018-20839, CVE-2018-15688, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

  EXTRA-HIGH (7)

  -----

  CVE-2019-3462 (apt@1.2.35.UBUNTU)

  CVE-2016-7543 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2020-6096 (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-6096 (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2019-11922 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2017-17522 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2017-17522 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  HIGH (18)

  -----

  CVE-2018-6557 (base-files@9.4ubuntu4.13.UBUNTU)

  CVE-2019-18276, CVE-2017-5932, CVE-2016-0634 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3549, CVE-2021-20294, 73 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2015-8865, CVE-2014-9653 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2021-3345, CVE-2019-14855, 2 more... (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2018-5733, CVE-2018-5732, CVE-2017-3144 (isc-dhcp-client@4.3.3-5ubuntu12.10.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2016-6305 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2018-6952, CVE-2018-6951, CVE-2018-20969 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2016-2381 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1125, CVE-2018-1124, 2 more... (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2019-20916 (python3-pip@8.1.1-2ubuntu0.6.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-1712, CVE-2019-3844, 8 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2016-6321 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2018-7738 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  MEDIUM (18)

  -----

  CVE-2018-0501, CVE-2016-1252 (apt@1.2.35.UBUNTU)

  CVE-2016-9401 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3487, CVE-2021-20284, 80 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-18018 (coreutils@8.25-2ubuntu3~16.04.UBUNTU)

  CVE-2017-1000249, CVE-2014-9621, CVE-2014-9620 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2011-2207 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2019-20795 (iproute2@4.3.0-1ubuntu3.16.04.5.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-24032, CVE-2021-24031 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2018-0739, CVE-2016-6308, CVE-2016-6307 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2019-20633, CVE-2016-10713 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-13776, CVE-2020-13529, 3 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2021-20193 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2016-5011 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  CVE-2021-28363 requirements.txt (urllib3@1.26.0)

  LOW (4)

  -----

  USN-3156-0002 (apt@1.2.35.UBUNTU)

  CVE-2020-35448 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2019-1552 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  USN-4120-0002, CVE-2019-20386, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

Included in the list of vulnerabilities identified by MergeBase is the vulnerable version of urllib3 we installed.

To ensure we are comparing like for like, we should also run the MergeBase scan on the updated image after applying the Ubuntu updates as we did with Docker scan. We can reuse the same MergeBase command line after updating the relevant parameters.

java -jar mergebase.jar –mode=profile –name=updated_ubuntu_image updated_ubuntu_image:latest

Interestingly, the results for the updated Ubuntu image are identical before and after running the updates. 

We can deduct the following outcomes from this exercise:

  1. Docker Scan and MergeBase find different vulnerabilities on the same containers.
  2. MergeBase provides far more detailed results when it comes to the actual software components installed on a container. The text output from the command line tool and the report it generates even details the relevant CVEs. 
  3. The Docker scan results change after running system updates while Mergebase remains the same. This difference indicates that Docker scan focuses more on the operating system while MergeBase concentrates on its specialty, Software Composition Analysis.

Container Scanning Limitations

Our rudimentary experiment shows that default container scanning solutions do not find application-layer vulnerabilities. Consequently, even though these tools provide a critical service that helps you find security issues in your containerized application, they do not offer a complete solution. For example, if we build a Python app and use the vulnerable urllib3 library, our containerized app is not secure. In that instance, a container scan will not show any vulnerabilities, but an attacker could create a rogue certificate authority and compromise your containerized service.

However, this simple illustration is only one example. If we consider that researchers found 18,325 vulnerabilities in 2020 and 7,545 in the first five months of 2021, protecting your containerized application requires multiple layers of security. If you only rely on container scanning solutions, you are not getting the visibility you need for application-layer vulnerabilities, as our urllib3 example demonstrates. What you need is a layered defense-in-depth security approach. 

Defense in Depth for Containerized Applications

While containerized applications offer several benefits, such as application consistency, scalability, and cost-effectiveness, managing all the moving parts that provide the platform running your app can be challenging. Since hosts, images, and containers make up a standard Docker architecture, a vulnerability in any of these three critical components could lead to a security incident. 

For example, should a threat actor manage to compromise an image on Docker Hub, it could lead to a devastating supply chain attack. A vulnerability in your Docker host, whether you use a cloud platform, Windows, or Linux server, could also lead to a compromise of your container. Finally, there is the container itself. As we have illustrated with our simple example, standard container scanning apps do not find application-layer vulnerabilities. 

Implementing defense-in-depth security for your containerized application requires a holistic approach. First, you need to secure your platform from external threats by implementing the relevant network segmentation, firewalls, and similar technologies that prevent unauthorized access. Then you need to continuously harden your hosts, images, and containers by leveraging Docker container scanning and implementing patches and upgrades. 

The final element you need to secure in your defense-in-depth strategy is your application layer. Since firewalls and vulnerability scanners that focus on hosts, images, and containers do not identify any application-related security threats, you need a solution to identify and mitigate this risk. As our example illustrated, Software Composition Analysis (SCA) tools, like MergeBase, provide visibility into the real risk of enterprise applications. Since most, if not all, modern applications leverage external libraries and packages, identifying vulnerabilities in these elements is vital in securing your application. If we revisit our urllib3 example, scanning the containerized application with an SCA tool like MergeBase found the vulnerability, and 107 others the Docker scan failed to identify.

Ready to mitigate risks?

Start for Free!

Container Security Tools Comparison for Vulnerability Scans

Recently, we took on a new challenge: compare 5 popular container security tools, including our solution. We wanted to see how the products stack up against each other. How did they do? Read this full container security tools comparison to find out!

TL; DR:

Containers have been causing waves in IT and dev circles since 2013, when Docker’s container technology was launched. They have revolutionized deployment, adding both speed and stability and have become critical for most IT operations, so securing them is a priority for all of us. How well do various tools do that? See the table below:

Tool Step 1 (Squid) Step 2 (Patched) Step 3 (Add App) Result
MergeBase 3 1 2 ALL
Aqua 0 0 0 NONE
Snyk 2 0 0 2
Docker Hub 1 0 0 1
Quay 0 0 0 NONE
Container Security Tools Comparison Table

But before we get into the details, it’s worthwhile to quickly revisit the importance of application container security in the modern-day development landscape.

The Importance of Container Security

Containers enable developers to run applications quickly and reliably when moved from one computing environment to another. But despite their many advantages – including increased application isolation – containers also amplify security risks. Increasing adoption in production environments makes them attractive to malicious actors. Since traditional network security solutions cannot always protect against lateral attacks, a lot of effort goes into developing application container security solutions.

Container security refers to the tools (e.g. Docker container security solutions) and policies implemented to protect container integrity and reliability, mitigate risk, and minimize vulnerabilities.

Container Security Tools Compared

To protect containers from attacks, many security tools are available. Usually, they audit the Common Vulnerabilities and Exposures (CVE) set by the National Vulnerability Database (NVD), or the benchmarks set by the Center for Internet Security (CIS).
Most containerized applications and their underlying infrastructure are distributed widely and highly dynamic. In this scenario, manual vulnerability scanning can be time-consuming and resource-intensive. To reduce operational overhead, many tools offer automation. Some focus on specific aspects of the cloud-native ecosystem, e.g. runtime security.
For our analysis, we picked 5 popular automated container scanners:

Methodology

Before starting our analysis, we set up three images. The images are a logical progression where start with a vulnerable version of squid, then patch it and then add a vulnerable proprietary library, a proxy for applications you might produce and deploy in Docker images

container scanning expectations
  1. Seeded with a vulnerable version of Squid, a caching and forwarding HTTP web proxy
  2. Patch Squid to the latest safe version
  3. Download a proprietary jar file that is vulnerable. Your own applications would typically fall in this category, and it is challenging for most container scanning tools to analyze these.

We expected that each application would find vulnerabilities in all these steps.

However, this is not quite what happened!

Before we reveal the results of our tool comparison, here’s a sequence of steps that shows the Docker files used to build the images. Also, for readers planning to replicate our experiment, bear in mind that vulnerability scanning is sensitive to the date of the scan. We completed this experiment in early April 2021. New vulnerabilities may have been found and published since then, and security scanning tools themselves may have also changed.

Procedure to Build Images with Docker Files

Step 1 - Build vulnerable base image
Step 2 - Patch Vulnerability
Step 3 - Add Proprietary Library
The steps to build the Docker images used in this analysis.

If you need the build scripts, please ask us. We believe in transparency and are happy to provide them.

Results of Container Security Tools Comparison & Analysis

1. Aqua

For teams wondering how to secure Docker containers, Aqua claims to provide “enterprise-grade security for Docker environments” from development to production. Its tool scans images for vulnerabilities, malware, configuration issues, etc., for continuous image assurance. Its vulnerabilities database is aggregated from multiple, constantly-updated data streams to increase detection accuracy and provide better protection.

Aqua container scanning did not find any vulnerabilities

Despite these claims, the tool didn’t quite make the cut in our test. In fact, Aqua found no vulnerabilities at all, raising doubts about its effectiveness.

 

 

2. Snyk

Snyk helps teams automatically find, prioritize and fix vulnerabilities in containers throughout the container lifecycle. It can detect vulnerable dependencies during coding, prevent new vulnerabilities from passing through the build process, and test the production environment for newly-disclosed vulnerabilities.

Snyk says that it has fixed over 5 million container vulnerabilities. But during our tests, Snyk found two vulnerabilities in Step 1:

  • CVE-2020-25097 (Squid)
  • CVE-2021-30139 (apk-tools)
Step 1 - Snyk finds two vulnerabilities
Snyk Spots

Snyk did not find any vulnerabilities from Steps 2 and 3.

3. Docker Hub

Docker hub container scanning did find only one vulnerability

When a Docker image is pushed to Docker Hub, it automatically scans it for vulnerabilities. Teams can review the security state of images and fix identified issues for more secure deployments. The vulnerability report displays vulnerabilities and sorts them according to severity. It also displays information about the:

  • Package containing the vulnerability
  • Version in which it was introduced
  • Whether the vulnerability is fixed in a later version

In our analysis, we found that this Docker container security scanner is not effective at finding all vulnerabilities. During testing, it only found one vulnerability from Step 1.

4. Quay

Quay container scanning did not find any vulnerabilities

Quay automatically scans containers to provide a real-time view of known vulnerabilities. The scan report displays vulnerabilities by severity level: Low, Medium and High. It also specifies whether patches are available.

But in our vulnerability test, Quay found no vulnerabilities. For all 3 steps, the report displayed a “passed” status for the security scan.

And now, we have come to the final tool in our analysis: our own MergeBase tool.

5. MergeBase

In our analysis, only MergeBase found all vulnerabilities, including those the other tools missed:

MergeBase container scanning finds all vulnerabilities
  • CVE-2021-28116 (Squid), for which no patch is available
  • CVE-2016-5725 in the application, a directory traversal vulnerability in JCraft JSch before 0.1.54 on Windows, when the mode is ChannelSftp (source: CVE Mitre)

In summary, MergeBase found:

  • 3 vulnerabilities in Step 1
  • 1 vulnerability in Step 2
  • 2 vulnerabilities in Step 3

Container Security Tools Comparison Conclusion

In containerized environments, the deployment pipeline is often standardized across different dev teams. Container scanning can help find vulnerabilities and take proactive action to fix security gaps. Securing containers and building security into the CI/CD their pipeline can help reduce the size of the attack surface.

However, different container scanning solutions yield inconsistent results in the same environment. Worse, many solutions fall short of their claims to help strengthen end-to-end container security.

In our analysis of 5 application container security tools, we found that our tool, MergeBase, was the only one that could find all vulnerabilities in our testing environment. Thus, compared to other tools, MergeBase provides complete DevSecOps coverage and reliable container security.

Want to know more about MergeBase? Take a look here!

Discover More from MergeBase

Open Source Protection

Stay on top of the real risk of open source at any time.

Avoid false positives and get sophisticated upgrade guidance based on risk, compatibility and popularity.

More on Continuous Protection

Add RunTime Protection

Detect and defend against known-vulnerabilities at runtime. The only SCA to do so.

The quickest way to respond to an imminent threat like log4j with CVE-2021-44228.

More on Run-time Protection

Shift Left Now

CodeGreen is an early-warning defence for your in-house development and integrates directly into GitHub and BitBucket

More on BitBucket and Github apps