Blog

Software Bill of Materials (SBOM): A Guide to Software Supply Chain Security

The software bill of materials (SBOM) is a document that lists all the components and subcomponents of your software code. Some regulated industries require these documents as part of their software supply chain security efforts, and as third-party liability policies emerge, SBOMs are becoming a popular defense in the fight against cyber threats.

Cyber crime is currently one of the largest causes of criminal money damages in the world, costing individuals, companies, and governments more than $6 trillion USD in reported damages in 2021. This trend is on the rise, with large companies reporting cybercrime as the largest threat in terms of fraud in PwC’s 2022 Global Economic Crime and Fraud Survey.

pwcs global economic crime

Much of this crime involves exploiting third-party vulnerabilities: instead of outright hacking governments and enterprise organizations, hackers target the smaller code libraries that these larger entities license or build their applications upon. Because of this, governments are imposing new regulations to secure the software supply chain in certain critical industries.

Perhaps the most notable example of this is US President Biden’s 2021 executive order calling for heightened software supply chain security. This order includes the need for software vendors to provide key purchasers with a software bill of materials.

The SBOM is a relatively new concept, and practices haven’t become normalized yet. Because of this, we’ve put together a guide to help you understand what an SBOM is, what it’s for, and how to use it—whether you’re a software vendor or a customer.

NOTE: In addition to being an advanced SCA solution, MergeBase can help you produce and verify SBOM documents. If you’re not already using MergeBase, book a demo with us to see how we can help you secure your software supply chain today!

Software bill of materials (SBOM) defined

According to the US Cybersecurity and Infrastructure Security Agency, an SBOM is “a nested inventory, a list of ingredients that make up software components.”

Sounds a tad vague, doesn’t it? 

That’s because we know that in order to secure the software supply chain, both vendors and customers need to understand the components of the software applications they’re buying and selling. At the bare minimum, a software bill of materials should include a list of every third-party code library you implement in your application. To use the CISA’s “list of ingredients” metaphor, your software bill of materials should at least mention everything that you can “buy off the shelf”—or download for free, in the case of open source components. 

What does a minimum-viable SBOM include?

The US National Telecommunications and Information Administration (NTIA) lays out the required baseline attributes that a software bill of materials must entail, including information on third-party code as well as metadata on the SBOM itself.

Required SBOM metadata

Your SBOM should include the following attributes describing itself:

  • Author name: This is the person responsible for compiling the SBOM.
  • Timestamp: This marks when the SBOM was published.

Required third-party code attributes

Your SBOM should list all the third-party software you’ve built your application on, whether it’s paid or open source. For each of these software components, your SBOM should include:

  • Component name: This is the name of the third-party code package as specified by the supplier.
  • Supplier name: This credits the organization that supplied the third-party code (unless it’s already covered by the component name attribute).
  • Version string: This specifies which version of this third-party library you’re using.
  • Component relationship: This describes how the component fits into the entire SBOM. Your application has the “primary” relationship type, while the third-party components will have the “included in” relationship type. (Components may be included in other components.)
  • Unique identifiers: This provides any information that further identifies the component, helping your SBOM readers look up the component in relevant databases.

Commonly used SBOM formats

As of September 2021, SPDX is an internationally recognized ISO standard for creating software bills of materials, and the format is widely supported in the world of software supply chain security. 

In addition, CycloneDX (from OWASP) is often used by organizations whose operations involve SBOM generation and consumption. Some organizations prefer CycloneDX for its specificity, which makes it much easier for machines to read SBOMs in CycloneDX—while others prefer the more generally accepted SPDX standard. 

How SBOMs help software supply chain security efforts 

A comprehensive software bill of materials brings a host of benefits to both customers and vendors. By creating this document (and keeping it up to date), you can improve how you manage your software supply chain, vulnerabilities, sales, procurement, and quality assurance.

When done well, an SBOM creates a shared understanding between customers and vendors, allowing both parties to respond to vulnerabilities together quickly.

Software customers benefit from transparency

Because an SBOM lists all the ingredients in a vendor’s application, customers can streamline their due diligence processes when selecting a vendor. Customers can compare SBOMs, check for vulnerabilities using the right software composition analysis (SCA) tool, and better manage risks involved in selecting vendors.

A good SBOM also allows customers to more easily stay compliant with software supply chain security regulations. The transparency provided in an SBOM means that customers can monitor their vendors’ SBOMs with their own SCA tools, making it much easier for them to know when they’re vulnerable.

Furthermore, an SBOM gives customers more focus when patching vulnerabilities. Because their security teams have a well-defined set of components to keep secure, they can more easily stay focused on patching known vulnerabilities.

Software vendors can streamline enterprise sales

As regulations surrounding software supply chain security come into play, software companies will face more and more compliance barriers when selling to regulated entities (or even when selling to companies who feed into regulated entities). Sales conversations with enterprise prospects will always involve some security concerns, and having an SBOM ready to present can cut down on valuable sales time—either by proving that you’re a compliant vendor sooner or by surfacing a dealbreaker early in the conversation.

How to create an SBOM

If you are generating an SBOM as a vendor, the NTIA has provided a thorough playbook for SBOM generation to get you started. You can expect to follow these basic steps in building your SBOM:

  1. Make a list of third-party components implemented in your software. Your primary source of information for this step will be your own engineering team—but you can also use a software composition analysis (SCA) tool like MergeBase to generate this list.
  2. Gather the information required to fill out the baseline attributes listed above.
  3. Import this component data into a structured SBOM format.
  4. Validate the SBOM: check that the format works and all the information is correctly structured. 
  5. (Optional) Publish your SBOM so your customers and salespeople can reference it. 

Remember: You will need an SBOM for each version of your product—and every time you update one of your components, you will also need to update your SBOM.

How to obtain an SBOM from a vendor

If you’re required to obtain an SBOM from a vendor (or a hopeful vendor), you will need to follow these steps:

  1. Request an up-to-date SBOM from your vendor. Your vendor may have this publicly available, but it’s always good to double-check that you’re dealing with the correct SBOM for the application you’re using.
  2. Verify that SBOM using your SCA tool—our customers tell us that MergeBase is especially useful for double-checking this!

Remember: A single vendor may be managing many SBOMs. Not only does each software version require its own SBOM, but any custom builds will require their own SBOMs as well. A single vendor may have multiple SBOMs for the same product—and if they’re working with a large enterprise organization, they might even need several SBOMs for the same customer.

How to consume SBOMs in any format

As we’ve mentioned, the discipline of generating and consuming SBOMs is still immature. This means consuming SBOMs is a bit difficult today—most SBOM readers can only consume these documents if they were generated within the same system.

If you’re a software customer, we recommend using a system-agnostic SBOM reader. This will allow you to ingest SBOMs no matter what tool generated them. You could use a standalone reader, but an advanced SCA will take care of this for you.

(And yes, MergeBase is a system-agnostic reader: one of the benefits of using our SCA is that you can also consume SBOMs from any generator using the CycloneDX format!)

Common software supply chain security concerns about SBOMs

Supply chain security regulations will make SBOMs a norm in regulated industries—but because this is a new discipline, it’s understandable that some vendors have raised concerns about generating and publishing them. However, the prevailing belief is that the benefits of a good SBOM far outweigh the risks.

We’ve listed a few commonly voiced concerns people have regarding software bills of materials below, as well as our thoughts on how to address these concerns. If these concerns come up in supply chain security conversations at work, these responses can help put the concerned parties at ease.

An SBOM will tell hackers exactly where to attack us!”

Third-party vulnerabilities are the easiest way for hackers to infiltrate your otherwise secure system. Because an SBOM lists every single component and subcomponent used in your software, some are concerned that cyber criminals will simply target the weakest links in the supply chain.

However, most of the cybersecurity community agrees that the transparency an SBOM provides raises the accountability for both customers and vendors. Once a vulnerability is known, the whole supply chain can quickly patch it.

This level of transparency also incentivises vendors to source third-party code from the most secure suppliers they can, which in turn motivates the producers of subcomponents and sub-subcomponents to increase their own levels of security.

As an analogy, imagine an application is like a bank, and an SBOM is like a blueprint. While a blueprint might make it easier for a team of thieves to pull off a daring heist, it will also make it much easier for the bank’s security team to keep the vault safe: they know which parts of the building are most vulnerable, and they can allocate their security resources accordingly.

“An SBOM will give away our trade secrets!”

Just like a prize-winning baker wouldn’t want to give away her secret chocolate chip cookie recipe, some software vendors wince at the idea of publishing their own list of ingredients. However, if your software truly is an original work, this shouldn’t be an issue.

Sharing your list of ingredients shouldn’t be the same as sharing your recipe. An SBOM doesn’t display your proprietary code, and it doesn’t detail how your third-party code is implemented within your application. Unless your application is just a reskin of several other companies’ products with no added value, you should have nothing to worry about on this front.

“An SBOM will take too much time and effort to maintain!”

Making an SBOM once is easy. Keeping your SBOMs up to date and organized can get complicated, however. This will take extra time.

However, generating and maintaining SBOMs is preventive work. Flossing your teeth takes a little extra time out of each day. Changing your car’s oil takes a few hours out of each year. But compared to the costs and pains associated with a root canal or an engine failure, the benefits outweigh the costs. Similarly, the time it takes to keep your SBOMs current shouldn’t come anywhere close to the costs of time, damages, and loss of trust associated with an otherwise avoidable breach.

We recommend simply making SBOM generation and maintenance part of your routine: product development teams should include an SBOM as an artifact with every new build. It might take a little while to get used to this practice, but in the end, it will be worth it.

(This becomes very easy using MergeBase. You can generate a current SBOM document with the click of a button from our home screen. Plus, our command line tool allows you to automate SBOM production—which means MergeBase can automatically generate a new SBOM for you with every update!)

Create and verify SBOMs with MergeBase

As governments and industries increase software supply chain security, software bills of material will become more and more common. It’s smart to get ahead of the game and begin learning to create and consume SBOMs now. A great place to get started is the NTIA’s SBOM resource library: this includes how-to guides, explainer whitepapers, and even a few use cases for reference.

If you’re already a MergeBase customer, you can use our tools to generate your own SBOMs automatically—or import and export your suppliers’ SBOMs, even if you don’t have access to their code. 

If you’re not a current MergeBase customer, consider exploring the ways we can help you increase security against third-party vulnerabilities. We offer a full software composition analysis (SCA) suite that monitors your code libraries for known vulnerabilities, automatically detects and implements patches, and both generates and consumes SBOM documents. 

If you want to increase your software supply chain security, book a demo with us today and we’ll show you how it works!

What are the Best Tools for Generating SBOM (Software Bill Of Materials)?

image source: unsplash

Introduction

I needed some SBOM samples to test an upcoming MergeBase feature (we plan to offer SBOM import in addition to our existing SBOM export feature). So I grabbed 5 tools for generating SBOM, more or less randomly, and tried them out. Here’s what I found!

Methodology

I whipped up a small Java Maven project (see: log4j-transitive-example.git) that transitively depended on Log4J.  That way I could assess tools both for how well they generated SBOM when source code was available and when it wasn’t.  I also put together a small Docker image in which I manually added a single vulnerable “jsch-1.3.8.jar” library under /opt/jsch/, since a lot of these tools offer SBOM generation from Docker images, and I wanted to exercise this particular use case (software added using Docker’s “COPY” command instead of via package-management).  

With these samples ready, I essentially ran 3 tests against each SBOM generation tool:

  1. Generate SBOM from source code (the Log4J transitive project, pre-build).
  2. Generate SBOM from a deployed system (again, the Log4J transitive project, but post-build).
  3. Generate SBOM from a Docker image.

Here’s how each tool fared!

Generating SBOM – A Quick Bakeoff


CycloneDx – cyclonedx-maven-plugin

1. Can it generate SBOM from something I acquired (no source code)?
No.

2. Can it generate SBOM from something I built (full source code)?
Yes.

3. Can it generate SBOM from a Docker image?
No.

4. Are transitive dependencies included in the SBOM?
Yes, beautifully. Multi-module and maven-profiles are handled beautifully as well.

5. Ease Of Deployment
Difficult to start, but wonderful after that point! Imagine a car where it’s *very* difficult to open the door the first time, but once you’re in the driver’s seat, it drives beautifully and intuitively. You need to know how to edit your project’s Maven pom.xml to activate the CycloneDX generation for your project. The documentation was not great for a newcomer (it assumed Maven expertise). But once you got it working, the default behaviour generated excellent SBOM files.

6. Comments
CycloneDX founder (Steve Springett) is clearly deeply (and by that, I mean *deeply profoundly*) proficient with Maven and Java. The resulting SBOM is ideal (as good as is possible). Anyone building a Java/Maven project should immediately enable this and start including it in their builds! Looking at the git commit history for this project – yup, Mr. Springett is definitely to blame for a work of beauty here (commit-counts by author)!

    1 Author: aalzate
    1 Author: Gregory Anne
    1 Author: iabudiab
    1 Author: Jonas Arnold Clasen
    1 Author: Mirko Friedenhagen
    1 Author: Robert Klaus
    1 Author: Robert Varga
    3 Author: Hervé Boutemy
    4 Author: Prabhu Subramanian
    5 Author: Thomas Gaskell
    5 Author: M. Scott Ford
   55 Author: dependabot
  324 Author: Steve Springett

The problem, though, is that it requires a reasonable understanding of Maven to enable.

Note: Based on our own Maven and software-composition expertise, here at MergeBase we recommend the following configuration when using cyclonedx-maven-plugin:

                <includeCompileScope>true</includeCompileScope>
                <includeProvidedScope>false</includeProvidedScope>
                <includeRuntimeScope>true</includeRuntimeScope>
                <includeSystemScope>false</includeSystemScope>
                <includeTestScope>false</includeTestScope>

Syft (by Anchore)

1. Can it generate SBOM from something I acquired (no source code)?
Yes.

2. Can it generate SBOM from something I built (full source code)?
Yes, but not as good (e.g., for log4j-transitive sample, it only listed the single top-level dep (“twilio-8.6.1.jar”).

3. Can it generate SBOM from a Docker image?
Yes, but in its “Docker” mode it seems to only query the package-manager. It did not notice that I had copied in /opt/jsch/jsch-0.1.38.jar (no mention of “jsch” in the resulting SBOM).

4. Transitive Dependencies
Yes, but ONLY in the post-build directory scan mode, and as a flat list, not as interconnected relationships (this makes sense since they are all lying together on the disk together).

5. Ease of Deployment
Very easy.  But somewhat single-minded in its approach. By this, I mean this tool is very easy to operate, and it will do exactly what you tell it to do, but not necessarily what you wanted!  In a way, I appreciate Syft’s “just scan everything” philosophy. E.g., “syft -o json /” – it’s gonna go for it (scan my complete file-system from root).

This contrasts with cyclonedx-maven-plugin’s approach, which is more:  “if you configure me incorrectly, I will do nothing, but if you do manage to get me actually working, from then on I will provide perfect output, more perfect than you ever even knew or dreamed or thought possible.”

6.  Comments
I suspect Syft is looking at whatever it can find inside the files on disk (e.g., inside “commons-io.jar” it must be quickly considering ./META-INF/maven/commons-io/commons-io/pom.properties). I didn’t look at Syft’s internal logic, but the reason I think it’s working this way is because jsch-0.1.38.jar does not have a pom.properties inside, and so Syft messed it up ("bom-ref": "pkg:maven/jsch/jsch@0.1.38"). Assuming that’s a heuristic, that’s a damn fine heuristic, though! It would actually work for jsch-0.1.29 and older!

Nonetheless, Syft is a pretty amazing tool, especially considering it’s free/open-source.

One minor criticism – what’s with all these spurious “cpe23” entries in the JSON?

{
  "name": "syft:cpe23",
  "value": "cpe:2.3:a:apache_software_foundation:log4j:2.14.0:*:*:*:*:*:*:*"
},
{
  "name": "syft:cpe23",
  "value": "cpe:2.3:a:apache_software_foundation:api:2.14.0:*:*:*:*:*:*:*"
},
{
  "name": "syft:cpe23",
  "value": "cpe:2.3:a:Activator:log4j_api:2.14.0:*:*:*:*:*:*:*"
}

 

Microsoft (Microsoft.Sbom.Tool)

1. Can it generate SBOM from something I acquired (no source code)?
It depends on your definition of “SBOM.” Yes, this tool is willing to run “ls” or “dir” recursively and re-assemble the output into a file that is technically a valid SBOM file. But I need a “packages” section in my SBOM, and it didn’t create one. The “files” section it did create and fill with data is mostly useless for the problems SBOM is supposed to help solve.

2. Can it generate SBOM from something I built (full source code)?
Yes. This tool automatically realised that it needed to run “mvn dependency:tree” and it knew how to reformat that into a useful SBOM file, including a “packages” section

3. Can it generate SBOM from a Docker image?
Yes? No? I’m not sure. It says it can, and when I asked it to do this, it obviously did something and even correctly printed the number of Alpine packages in my Docker image. But there was no resulting SBOM file.

4. Transitive Dependencies
Ha! Very funny! No, they are not included in the output, although SPDX’s “dependency” feature *IS* used to list all the Maven references (but flattened). Like so:

{
  "relationshipType": "DEPENDS_ON",
  "relatedSpdxElement": "SPDXRef-Package-AC15BDA5FB1FE5FB5C52F9BB784F7FA5FDB04D590DFD57EC899C932579A8B4B1",
  "spdxElementId": "SPDXRef-RootPackage"
},
{
  "relationshipType": "DEPENDS_ON",
  "relatedSpdxElement": "SPDXRef-Package-E4453FF31A893CF22D5BDFCD71D0BB2A98C2554822A100A1AEB5C944E44DDF8D",
  "spdxElementId": "SPDXRef-RootPackage"
},

5. Ease of Deployment
Medium difficulty.  The tool requires a lot of command-line arguments, but the error messages and help-text guided me reasonably well.

6. Comments
I was disappointed by this tool, mainly for three reasons:

– Not including a “packages” section (when source code is not available) really defeats the point of SBOM.  So I do worry that people might be happily using this tool and thinking they are getting good SBOM files out of it, when they really are essentially getting garbage.

– Also, when the source code is available, I was a little disappointed that the full dependency-tree was flattened instead of keeping its original structure. But this was a relatively small disappointment – at least all the 3rd party libraries were accurately recorded!

– Finally, I was not able to generate SBOM for a Docker image, even though it seemed to happily process the image I provided.


 

Fossa

1. Can it generate SBOM from something I acquired (no source code)?
No (or at least I could not find any way to do it!).

2. Can it generate SBOM from something I built (full source code)?
Yes.

3. Can it generate SBOM from a Docker image?
Yes, but on the paid plan (I only tried the free plan).

4. Are transitive dependencies included in the SBOM?
Yes, but on the paid plan (I only tried the free plan).

5. Ease Of Deployment
Excellent.

6. Comments
I was surprised to see they use SPDX’s plain-text format (not XML and not JSON):

SPDXVersion: SPDX-2.1
DataLicense: CC0-1.0
SPDXID: SPDXRef-DOCUMENT
Creator: Organization: FOSSA, INC.
Created: 2022-08-09T10:45:17Z

Unfortunately, the generated SBOM file was not particularly useful. This is not really a useful way to identify this library (uhh, natural language?) !!!

PackageName: Apache Log4j API
PackageVersion: 2.14.0

I would much prefer something that could be be mapped back to Maven-Central coordinates (e.g., repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.14.0/ or, if you prefer, groupId=org.apache.logging and artifactId=log4j-api) since both of these are unambiguous and both are essentially how developers think and talk about these dependencies in the first place! Another great option: use PURLs!

Finally, the license info included in the SBOM output made no sense to me:

PackageLicenseDeclared: Apache-2.0
PackageCopyrightText: ownership.
PackageLicenseInfoFromFiles: Apache-2.0
PackageLicenseInfoFromFiles: MIT
PackageLicenseInfoFromFiles: Apache-1.1

I quickly ran my own manual analysis of “Log4J-API-2.14.0” to verify these claimed licenses (note: I’m not a lawyer, but I have published academic papers on this problem, and I’m also an Apache committer). I could find ZERO pieces of evidence suggesting that MIT or Apache-1.1 were remotely appropriate here. Flagging “Apache-1.1” as a possible license here, while not only inaccurate, could also cause bigger problems since Apache-1.1 is famously incompatible when combined with GPL software (whereas MIT is a benign and highly cross-compatible license).


MergeBase

1. Can it generate SBOM from something I acquired (no source code)?
Yes. MergeBase is able to analyse binary applications, including Java and .NET and subsequently generate an SBOM with the click of button, or in an automated workflow.

2. Can it generate SBOM from something I built (full source code)?
Yes. Same as 1

3. Can it generate SBOM from a Docker image?
Yes.

4. Are transitive dependencies included in the SBOM?
Yes.

5. Ease Of Deployment
Excellent.

6. Comments

Not only is MergeBase the most comprehensive SCA tool to create SBOM’s with, it also is able to import SBOM’s and automatically do a full risk analysis on it for security, legal and technical (debt) risks.


Snyk (via snyk2spdx optional tool)

1. Can it generate SBOM from something I acquired (no source code)?
See answer #2.

2. Can it generate SBOM from something I built (full source code)?
I guess technically the result is a valid SBOM file….

3. Can it generate SBOM from a Docker image?
Again, see answer #2 (above).

4. Are transitive dependencies included in the SBOM?
Yes, except Snyk’s SBOM implementation here has a game-over flaw (see answer #2, and see comments below).

5. Ease Of Deployment
Excellent:  npm install -g snyk2spdx.

6. Comments
One has to wonder what they were thinking! Yes the end-result is a valid SBOM file, but it lacks a “packages” section.  It even lacks the useless “files” section.

What does it have instead?  A “vulnerabilities” section !!!

To bring this back to SBOM’s original shipping metaphor (a “bill of materials”), this would be like asking IKEA to send you the list of parts for a particular bunkbed. And IKEA answering with:  “We looked up all the parts for the bunkbed. Please find enclosed a description of 3 of those parts (out of an unknown total). These particular 3 parts contributed to serious bunkbed failures in the last 5 years. We’ve included descriptions of how they failed for your convenience.  Enjoy your SBOM!”

If we are not understanding this tool, or if Snyk has a different, more appropriate SBOM offering, I would love an email from Snyk to help me understand this better (you can reach me at julius@mergebase.com).

Update from Snyk!

Gareth Rushgrove (VP, Product Management) sent me an email. In his own words:

The snyk2spdx tool is not meant to generate an SBOM. It was an experiment looking at the WIP vulnerability extension in the draft SPDX 3.0 spec. We built the snyk2spdx tool to kick the tires there. Technically it’s generating a basic VEX.

This is very helpful information – thank you Snyk! This is a good point: I mistakenly assumed that “snyk2spdx” was an SBOM generator (since SBOM is SPDX’s primary use case these days), but SPDX can be used for other purposes.

Gareth also mentioned that Snyk is working on bringing SBOM generation to their main “snyk” command-line tool, and that they are also piloting an ability right now to work with CycloneDX data directly from Git repositories. Working directly from Git repositories (without invoking build tools) is a whole other kettle of fish that I did not touch on here! Very exciting!


 

Conclusion

If you can, use CycloneDX native plugins for your software systems. These produce perfect SBOMs for your software that you can confidently share with your customers. But you must remember to enable CycloneDX for each language in your software (which can require a bit of work).  

However, if you do not have access to source code, we recommend Syft (free) or MergeBase (free trial), since both of these products are able to produce accurate, useful SBOMs when source code is not available.

When generating SBOMs from Docker images, it’s very important to understand that many SBOM generators tend to only query the package-management system, and so Docker-derived SBOMs will probably usually miss the most important part of the image – the actual software you are trying to run inside Docker!

Doesn’t have access to source code and want to generate an SBOM?

When Dependabot Is Worse Than Nothing: Log4J As A Sub-Dependency

Watch if this webinar about Dependabot applies to you and find out how to fix this :).

from the “Webinar Wednesday  from March 30th, 2022, with Jim and Julius

Why should you care about using Dependabot?

Because if you’re using industry-standard software leader Dependabot, then your devs didn’t fix the recent Log4J problem properly.

If you’re using it, then the tools you’re using now aren’t getting the job done.

In practice, it has a serious implementation flaw: it can only see transitive dependencies (aka sub-dependencies) in languages and dependency managers that support lock files.

Dependabot: Theory vs Reality

In theory, Dependabot is exactly what the world needs to keep software dependency chains safe from known vulnerabilities: tightly integrated with Github; auto-generates pull requests; plugged into Github Security Advisories (GHSA); it also supports a wide range of programming languages and dependency managers.

But in practice it has a serious implementation flaw: it can only see transitive dependencies (aka sub-dependencies) in languages and dependency managers that support lock files.

Do you know any languages that currently DO NOT support lock files?

Java / Maven!

This has some bad implications if you’re using it to protect yourself from Log4J (since Log4J is a Java library).

Want to know more?

Ubuntu Docker Container Scanning – Is it enough to protect your app?

In this post, we are going to look at Docker container scanning:

  1. First, we will build an Ubuntu Docker container, add some software to it, run a Docker scan, and record the vulnerabilities.
  2. Then, we will try to resolve some of those vulnerabilities, run another scan, and examine the results.
  3. Finally, we will determine whether container scanning is enough to protect your applications and data from compromise. 

Building the Docker Container

We will build our Ubuntu Docker container using the Ubuntu Xenial image. Since it is a few versions behind, it will hopefully contain a few vulnerabilities. We are also going to install Python, PIP3, and a vulnerable version of urllib3. If we search the CVE database, we find CVE-2021-28363. According to this CVE, the urllib3 library 1.26.x before 1.26.4 omits SSL certificate validation in some cases involving HTTPS to HTTPS proxies.

The first step in our process is to pull the Ubuntu Xenial Docker image from Docker Hub and run it as a container on our Docker host. The command below will get the image, start the container in interactive mode, and name it ubuntu_xenial. 

docker run -it –name ubuntu_xenial ubuntu:xenial

Once the process completes, you will be presented with the Ubuntu prompt, as shown in the text output below.

PS C:\Users\docker> docker run -it --name ubuntu_xenial ubuntu:xenial

Unable to find image 'ubuntu:xenial' locally

xenial: Pulling from library/ubuntu

80bce60046fa: Pull complete

55a738a15540: Pull complete

e19cf0706c62: Pull complete

de4cdd6c27d1: Pull complete

Digest: sha256:9775877f420d453ef790e6832d77630a49b32a92b7bedf330adf4d8669f6600e

Status: Downloaded newer image for ubuntu:xenial

root@a317929cf5d7:/#

The next step is to install Python and PIP by running the following command:

apt update && apt install python3-pip

root@a317929cf5d7:/# apt update && apt install python3-pip

Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]

Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]

Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]

Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [2051 kB]
.....
Fetched 19.4 MB in 42s (458 kB/s)

Reading package lists... Done

Building dependency tree

Reading state information... Done

5 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done
.....
The following packages will be upgraded:

libc6

1 upgraded, 85 newly installed, 0 to remove and 4 not upgraded.

Need to get 98.7 MB of archives.

After this operation, 289 MB of additional disk space will be used.

Do you want to continue? [Y/n]

 

After installing Python and PIP, we are ready to install urllib3. However, since the development team fixed the vulnerability in version 1.26.4, we need to install an earlier vulnerable version. For the purposes of this exercise, let’s install version 1.26.0 from requirements.txt. 

We can check that our requirements.txt file includes our vulnerable version by running the cat command.

root@a317929cf5d7:/# cat requirements.txt

urllib3==1.26.0

We can use PIP to install the  requirements.txt file.

pip3 install -r requirements.txt

root@a317929cf5d7:/# pip3 -r requirements.txt

Collecting urllib3==1.26.0

  Downloading https://files.pythonhosted.org/packages/7e/a7/746338eb8addda2e7662ee5e10a9f85150aba013cd610c9569c17146b914/urllib3-1.26.0-py2.py3-none-any.whl (136kB)

    100% |################################| 143kB 3.9MB/s

Installing collected packages: urllib3

Successfully installed urllib3-1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

root@a317929cf5d7:/urllib3_test# pip3 freeze requirements.txt

urllib3==1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

Now that we have updated our container, it is time to commit it and create our image by running the following Docker command from the docker host.

docker commit ubuntu_xenial

PS C:\Users\docker> docker commit ubuntu_xenial

sha256:130de6a9e3822dd3bbf19131469f3de441967756cd6faf70f52c92183ddea37e

Let’s check that Docker created the image by running the docker images command.

docker images

As we can see in the text output below, Docker created the image. 

PS C:\Users\docker> docker images

REPOSITORY   TAG      IMAGE ID      CREATED              SIZE

<none>      <none>    130de6a9e382   About a minute ago   465MB

ubuntu       xenial   9ff95a467e45   12 days ago          135MB

However, to make our lives easier, let’s give the new image a tag to identify it by running the following command.

docker tag 130de6a9e382 our_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag 130de6a9e382 our_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED        SIZE

our_ubuntu_image  latest   130de6a9e382   3 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago     135MB

As Docker’s scanning feature only works on images and not containers, we are now ready to scan the image for any vulnerabilities. Let’s start the Docker container scanning by running the following command.

docker scan our_ubuntu_image

PS C:\Users\docker> docker scan our_ubuntu_image

Docker Scan relies upon access to Snyk, a third party provider, do you consent to proceed using Snyk? (y/N)docker y

/ Analyzing container dependencies for our_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 193 issues, as shown in the text output below.

Package manager:  deb

Project name:     docker-image|our_ubuntu_image

Docker image:     our_ubuntu_image

Platform:         linux/amd64

Licenses:        enabled

Tested 185 dependencies for known issues, found 193 issues.

Docker Container Scanning Results

If we study the vulnerability report, we find that Docker container scanning highlights some vulnerabilities that can be fixed, as shown in the text output below.

✗  Low severity vulnerability found in glibc/libc-bin

  Description: Improper Data Handling

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-345675

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

✗ Low severity vulnerability found in glibc/libc-bin

  Description: Integer Underflow

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-571388

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

If we look at the message for the two vulnerabilities in the image, we need to update those particular packages to resolve the issue. First, we need to create a container from the new image to make sure we are working with the latest version. As we did previously, we need to execute the docker run command as shown in the image below.

docker run -it –name our_ubuntu_image our_ubuntu_image

PS C:\Users\docker> docker run -it --name our_ubuntu_image our_ubuntu_image

root@0dfcbff2e3d1:/#

After executing the docker run command, we can run the following commands to update Ubuntu from the container’s command line.

apt update && apt upgrade

root@0dfcbff2e3d1:/# apt update && apt upgrade

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease

Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease

Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease

Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease

Reading package lists... Done

Building dependency tree

Reading state information... Done

4 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

The following packages will be upgraded:

  apt libapt-pkg5.0 libc-bin multiarch-support

4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Need to get 2458 kB of archives.

After this operation, 70.7 kB of additional disk space will be used.

Do you want to continue? [Y/n]

After running and installing the updates, we need to commit and create another Docker image to save the changes. 

docker commit our_ubuntu_image

PS C:\Users\docker> docker commit our_ubuntu_image

sha256:bf5809ae94d91ea37ac94fcc7260d607d15925347267bba911e9d3e2308dd93f

We can check that Docker created the image successfully by running the docker images command.

docker images

As we can see in the text output below, Docker confirms the new updated image exists.

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED          SIZE

<none>             <none>    bf5809ae94d9   2 minutes ago    476MB

our_ubuntu_image   latest    130de6a9e382   19 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago      135MB

As we did before, let’s give the new image a tag to identify it by running the following command.

docker tag bf5809ae94d9 updated_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag bf5809ae94d9 updated_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY             TAG       IMAGE ID       CREATED          SIZE

updated_ubuntu_image   latest    bf5809ae94d9   3 minutes ago   476MB

our_ubuntu_image       latest    130de6a9e382   21 minutes ago 465MB

ubuntu                 xenial    9ff95a467e45   12 days ago       135MB

We are now ready to scan the updated container to see if our efforts resolved any vulnerabilities. As before, to run docker scan execute the following command. Note that we are now scanning our updated image.

docker scan updated_ubuntu_image

PS C:\Users\docker> docker scan updated_ubuntu_image

/ Analyzing container dependencies for updated_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 191 issues, as shown in the text output below. This result demonstrates that updating our Ubuntu container resolved two vulnerabilities. 

Package manager:    deb

Project name:      docker-image|updated_ubuntu_image

Docker image:      updated_ubuntu_image

Platform:          linux/amd64

Licenses:         enabled

Tested 185 dependencies for known issues, found 191 issues.

However, if we search both the original vulnerability report and the report generated after our updates, neither report contains an alert for CVE-2021-28363. If you recall, when we created our image, we installed a vulnerable version of urllib3. Since neither scan picked up this vulnerability, we need to consider alternatives that mitigate this risk.

MergeBase Vulnerability Analysis and Resolution

To test the validity of Docker scan’s analysis and reporting, let’s run the same scans using MergeBase.

As in the first example, we first need to run a scan on the Docker image named our_ubuntu_image before we run any Ubuntu updates. 

First, we need to get the docker image repository and tag by running the following command:

docker images

As we can see in the text output below, the repository name is ‘our_ubuntu_image,’ and its tag is ‘latest.’

REPOSITORY TAG IMAGE ID CREATED SIZE

our_ubuntu_image latest 130de6a9e382 18 minutes ago 462MB

ubuntu xenial 9ff95a467e45 3 weeks ago 135MB

Using the MergeBase command-line tool, we can scan this image using the following command:

java -jar mergebase.jar –mode=profile –name=our_ubuntu_image our_ubuntu_image:latest

If we look at the summary scan report on the MergeBase portal, we can see that it found 301 vulnerabilities. This number exceeds the 193 found by Docker scan by 108 vulnerabilities.

Below is the text output from the command-line tool. As you can see, it details multiple CVEs and categorizes them by severity.

java -jar mergebase.jar --mode=profile --name=updated_ubuntu_image updated_ubuntu_image:latest

Saving application profile to https://demo.mergebase.com

  Vulnerable Files Overview

  =========================

  CRITICAL (12)

  -----

  CVE-2017-7614, CVE-2017-7226, CVE-2017-6969 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-8283 (dpkg-dev@1.18.4ubuntu1.7.UBUNTU)

  CVE-2017-8283 (dpkg@1.18.4ubuntu1.7.UBUNTU)

  CVE-2019-18218 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2008-1530 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2016-6309 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2017-12814 (perl-base@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2017-12814 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1126 (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2018-20839, CVE-2018-15688, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

  EXTRA-HIGH (7)

  -----

  CVE-2019-3462 (apt@1.2.35.UBUNTU)

  CVE-2016-7543 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2020-6096 (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-6096 (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2019-11922 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2017-17522 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2017-17522 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  HIGH (18)

  -----

  CVE-2018-6557 (base-files@9.4ubuntu4.13.UBUNTU)

  CVE-2019-18276, CVE-2017-5932, CVE-2016-0634 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3549, CVE-2021-20294, 73 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2015-8865, CVE-2014-9653 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2021-3345, CVE-2019-14855, 2 more... (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2018-5733, CVE-2018-5732, CVE-2017-3144 (isc-dhcp-client@4.3.3-5ubuntu12.10.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2016-6305 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2018-6952, CVE-2018-6951, CVE-2018-20969 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2016-2381 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1125, CVE-2018-1124, 2 more... (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2019-20916 (python3-pip@8.1.1-2ubuntu0.6.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-1712, CVE-2019-3844, 8 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2016-6321 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2018-7738 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  MEDIUM (18)

  -----

  CVE-2018-0501, CVE-2016-1252 (apt@1.2.35.UBUNTU)

  CVE-2016-9401 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3487, CVE-2021-20284, 80 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-18018 (coreutils@8.25-2ubuntu3~16.04.UBUNTU)

  CVE-2017-1000249, CVE-2014-9621, CVE-2014-9620 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2011-2207 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2019-20795 (iproute2@4.3.0-1ubuntu3.16.04.5.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-24032, CVE-2021-24031 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2018-0739, CVE-2016-6308, CVE-2016-6307 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2019-20633, CVE-2016-10713 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-13776, CVE-2020-13529, 3 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2021-20193 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2016-5011 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  CVE-2021-28363 requirements.txt (urllib3@1.26.0)

  LOW (4)

  -----

  USN-3156-0002 (apt@1.2.35.UBUNTU)

  CVE-2020-35448 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2019-1552 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  USN-4120-0002, CVE-2019-20386, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

Included in the list of vulnerabilities identified by MergeBase is the vulnerable version of urllib3 we installed.

To ensure we are comparing like for like, we should also run the MergeBase scan on the updated image after applying the Ubuntu updates as we did with Docker scan. We can reuse the same MergeBase command line after updating the relevant parameters.

java -jar mergebase.jar –mode=profile –name=updated_ubuntu_image updated_ubuntu_image:latest

Interestingly, the results for the updated Ubuntu image are identical before and after running the updates. 

We can deduct the following outcomes from this exercise:

  1. Docker Scan and MergeBase find different vulnerabilities on the same containers.
  2. MergeBase provides far more detailed results when it comes to the actual software components installed on a container. The text output from the command line tool and the report it generates even details the relevant CVEs. 
  3. The Docker scan results change after running system updates while Mergebase remains the same. This difference indicates that Docker scan focuses more on the operating system while MergeBase concentrates on its specialty, Software Composition Analysis.

Container Scanning Limitations

Our rudimentary experiment shows that default container scanning solutions do not find application-layer vulnerabilities. Consequently, even though these tools provide a critical service that helps you find security issues in your containerized application, they do not offer a complete solution. For example, if we build a Python app and use the vulnerable urllib3 library, our containerized app is not secure. In that instance, a container scan will not show any vulnerabilities, but an attacker could create a rogue certificate authority and compromise your containerized service.

However, this simple illustration is only one example. If we consider that researchers found 18,325 vulnerabilities in 2020 and 7,545 in the first five months of 2021, protecting your containerized application requires multiple layers of security. If you only rely on container scanning solutions, you are not getting the visibility you need for application-layer vulnerabilities, as our urllib3 example demonstrates. What you need is a layered defense-in-depth security approach. 

Defense in Depth for Containerized Applications

While containerized applications offer several benefits such as application consistency, scalability, and cost-effectiveness, managing all the moving parts that provide the platform running your app can be challenging. Since hosts, images, and containers make up a standard Docker architecture, a vulnerability in any of these three critical components could lead to a security incident. 

For example, should a threat actor manage to compromise an image on Docker Hub, it could lead to a devastating supply chain attack. A vulnerability in your Docker host, whether you use a cloud platform, Windows, or Linux server, could also lead to a compromise of your container. Finally, there is the container itself. As we have illustrated with our simple example, standard container scanning apps do not find application-layer vulnerabilities. 

Implementing defense-in-depth security for your containerized application requires a holistic approach. First, you need to secure your platform from external threats by implementing the relevant network segmentation, firewalls, and similar technologies that prevent unauthorized access. Then you need to continuously harden your hosts, images, and containers by leveraging Docker container scanning and implementing patches and upgrades. 

The final element you need to secure in your defense-in-depth strategy is your application layer. Since firewalls, and vulnerability scanners that focus on hosts, images, and containers, do not identify any application-related security threats, you need a solution to identify and mitigate this risk. As our example illustrated, Software Composition Analysis (SCA) tools like MergeBase, provide visibility into the real risk of enterprise applications. Since most, if not all, modern applications leverage external libraries and packages, identifying vulnerabilities in these elements is vital in securing your application. If we revisit our urllib3 example, scanning the containerized application with an SCA tool like MergeBase found the vulnerability and 107 others the Docker scan failed to identify.

Ready to mitigate risks?

Container Security Tools Comparison for Vulnerability Scans

Recently, we took on a new challenge: compare 5 popular container security tools, including our solution. We wanted to see how the products stack up against each other. How did they do? Read this full container security tools comparison to find out!

TL; DR:

Containers have been causing waves in IT and dev circles since 2013 when Docker’s container technology was launched. They have revolutionized deployment, adding both speed and stability and have become critical for most IT operations, so securing them is a priority for all of us. How well do various tools do that? See the table below:

ToolStep 1 (Squid)Step 2 (Patched)Step 3 (Add App)Result
MergeBase312ALL
Aqua000NONE
Snyk2002
Docker Hub1001
Quay000NONE
Container Security Tools Comparison Table

But before we get into the details, it’s worthwhile to quickly revisit the importance of application container security in the modern-day development landscape.

The Importance of Container Security

Containers enable developers to run applications quickly and reliably when moved from one computing environment to another. But despite their many advantages – including increased application isolation – containers also amplify security risks. Increasing adoption in production environments makes them attractive to malicious actors. Since traditional network security solutions cannot always protect against lateral attacks, a lot of effort goes into developing application container security solutions.

Container security refers to the tools (e.g. Docker container security solutions) and policies implemented to protect container integrity and reliability, mitigate risk, and minimize vulnerabilities.

Container Security Tools Compared

To protect containers from attacks, many security tools are available. Usually, they audit the Common Vulnerabilities and Exposures (CVE) set by the National Vulnerability Database (NVD), or the benchmarks set by the Center for Internet Security (CIS).
Most containerized applications and their underlying infrastructure are distributed widely and highly dynamic. In this scenario, manual vulnerability scanning can be time-consuming and resource-intensive. To reduce operational overhead, many tools offer automation. Some focus on specific aspects of the cloud-native ecosystem, e.g. runtime security.
For our analysis, we picked 5 popular automated container scanners:

Methodology

Before starting our analysis, we set up three images. The images are a logical progression where start with a vulnerable version of squid, then patch it and then add a vulnerable proprietary library, a proxy for applications you might produce and deploy in Docker images

container scanning expectations
  1. Seeded with a vulnerable version of Squid, a caching and forwarding HTTP web proxy
  2. Patch Squid to latest safe version
  3. Download a proprietary jar file that is vulnerable. Your own applications would typically fall in this category and it is challenging for most container scanning tools to analyze these.

We expected that each application would find vulnerabilities in all these steps.

However, this is not quite what happened!

Before we reveal the results of our tool comparison, here’s a sequence of steps that shows the Docker files used to build the images. Also, for readers planning to replicate our experiment, bear in mind that vulnerability scanning is sensitive to the date of the scan. We completed this experiment in early April 2021. New vulnerabilities may have been found and published since then, and security scanning tools themselves may have also changed.

Procedure to Build Images with Docker Files

If you need the build scripts, please ask us. We believe in transparency and are happy to provide them.

Results of Container Security Tools Comparison & Analysis

1. Aqua

For teams wondering how to secure Docker containers, Aqua claims to provide “enterprise-grade security for Docker environments” from development to production. Its tool scans images for vulnerabilities, malware, configuration issues, etc. for continuous image assurance. Its vulnerabilities database is aggregated from multiple, constantly-updated data streams to increase detection accuracy and provide better protection.

Aqua container scanning did not find any vulnerabilities

Despite these claims, the tool didn’t quite make the cut in our test. In fact, Aqua found no vulnerabilities at all, raising doubts about its effectiveness.

2. Snyk

Snyk helps teams automatically find, prioritize and fix vulnerabilities in containers throughout the container lifecycle. It can detect vulnerable dependencies during coding, prevent new vulnerabilities from passing through the build process, and test the production environment for newly-disclosed vulnerabilities.

Snyk says that it has fixed over 5 million container vulnerabilities. But during our tests, Snyk found two vulnerabilities in Step 1:

  • CVE-2020-25097 (Squid)
  • CVE-2021-30139 (apk-tools)

Snyk did not find any vulnerabilities from Steps 2 and 3.

3. Docker Hub

Docker hub container scanning did find only one vulnerability

When a Docker image is pushed to Docker Hub, it automatically scans it for vulnerabilities. Teams can review the security state of images, and fix identified issues for more secure deployments. The vulnerability report displays vulnerabilities, and sorts them according to severity. It also displays information about the:

  • Package containing the vulnerability
  • Version in which it was introduced
  • Whether the vulnerability is fixed in a later version

In our analysis, we found that this Docker container security scanner is not effective at finding all vulnerabilities. During testing, it only found one vulnerability from Step 1.

4. Quay

Quay container scanning did not find any vulnerabilities

Quay automatically scans containers to provide a real-time view of known vulnerabilities. The scan report displays vulnerabilities by severity level: Low, Medium and High. It also specifies whether patches are available.

But in our vulnerability test, Quay found no vulnerabilities. For all 3 steps, the report displayed a “passed” status for the security scan.

And now, we come to the final tool in our analysis: our own MergeBase tool.

5. MergeBase

In our analysis, only MergeBase found all vulnerabilities, including those the other tools missed:

MergeBase container scanning finds all vulnerabilities
  • CVE-2021-28116 (Squid) for which no patch is available
  • CVE-2016-5725 in the application, a directory traversal vulnerability in JCraft JSch before 0.1.54 on Windows, when the mode is ChannelSftp (source: CVE Mitre)

In summary, MergeBase found:

  • 3 vulnerabilities in Step 1
  • 1 vulnerability in Step 2
  • 2 vulnerabilities in Step 3

Container Security Tools Comparison Conclusion

In containerized environments, the deployment pipeline is often standardized across different dev teams. Container scanning can help find vulnerabilities and take proactive action to fix security gaps. Securing containers and building security into the CI/CD their pipeline can help reduce the size of the attack surface.

However, different container scanning solutions yield inconsistent results in the same environment. Worse, many solutions fall short of their claims to help strengthen end-to-end container security.

In our analysis of 5 application container security tools, we found that our tool MergeBase was the only one that could find all vulnerabilities in our testing environment. Thus, compared to other tools, MergeBase provides complete DevSecOps coverage and reliable container security.

Want to know more about MergeBase? Take a look here!

Discover More from MergeBase

Open Source Protection

Stay on top of the real risk of open source at any time.

Avoid false positives and get sophisticated upgrade guidance based on risk, compatibility and popularity.

More on Continuous Protection

Add RunTime Protection

Detect and defend against known-vulnerabilities at runtime. The only SCA to do so.

The quickest way to respond to an imminent threat like log4j with CVE-2021-44228.

More on Run-time Protection

Shift Left Now

CodeGreen is an early-warning defence for your in-house development and integrates directly into GitHub and BitBucket

More on BitBucket and Github apps