In this post, we are going to look at Docker container scanning:
- First, we will build an Ubuntu Docker container, add some software to it, run a Docker scan, and record the vulnerabilities.
- Then, we will try to resolve some of those vulnerabilities, run another scan, and examine the results.
- Finally, we will determine whether container scanning is enough to protect your applications and data from compromise.
Building the Docker Container
We will build our Ubuntu Docker container using the Ubuntu Xenial image. Since it is a few versions behind, it will hopefully contain a few vulnerabilities. We are also going to install Python, PIP3, and a vulnerable version of urllib3. If we search the CVE database, we find CVE-2021-28363. According to this CVE, the urllib3 library 1.26.x before 1.26.4 omits SSL certificate validation in some cases involving HTTPS to HTTPS proxies.
The first step in our process is to pull the Ubuntu Xenial Docker image from Docker Hub and run it as a container on our Docker host. The command below will get the image, start the container in interactive mode, and name it ubuntu_xenial.
docker run -it –name ubuntu_xenial ubuntu:xenial
Once the process completes, you will be presented with the Ubuntu prompt, as shown in the text output below.
PS C:\\Users\\docker> docker run -it --name ubuntu\_xenial ubuntu:xenial Unable to find image 'ubuntu:xenial' locally xenial: Pulling from library/ubuntu 80bce60046fa: Pull complete 55a738a15540: Pull complete e19cf0706c62: Pull complete de4cdd6c27d1: Pull complete Digest: sha256:9775877f420d453ef790e6832d77630a49b32a92b7bedf330adf4d8669f6600e Status: Downloaded newer image for ubuntu:xenial root@a317929cf5d7:/#
The next step is to install Python and PIP by running the following command:
_apt update && apt install python3-pip_
root@a317929cf5d7:/# apt update && apt install python3-pip Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease \[247 kB\] Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease \[109 kB\] Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease \[109 kB\] Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages \[2051 kB\] ..... Fetched 19.4 MB in 42s (458 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 5 packages can be upgraded. Run 'apt list --upgradable' to see them. Reading package lists... Done Building dependency tree Reading state information... Done ..... The following packages will be upgraded: libc6 1 upgraded, 85 newly installed, 0 to remove and 4 not upgraded. Need to get 98.7 MB of archives. After this operation, 289 MB of additional disk space will be used. Do you want to continue? \[Y/n\]
After installing Python and PIP, we are ready to install urllib3. However, since the development team fixed the vulnerability in version 1.26.4, we need to install an earlier vulnerable version. For the purposes of this exercise, let’s install version 1.26.0 from requirements.txt.
We can check that our requirements.txt file includes our vulnerable version by running the cat command.
root@a317929cf5d7:/# cat requirements.txt urllib3==1.26.0
We can use PIP to install the requirements.txt file.
pip3 install -r requirements.txt
root@a317929cf5d7:/# pip3 -r requirements.txt Collecting urllib3==1.26.0 Downloading https://files.pythonhosted.org/packages/7e/a7/746338eb8addda2e7662ee5e10a9f85150aba 013cd610c9569c17146b914/urllib3-1.26.0-py2.py3-none-any.whl (136kB) 100% |################################| 143kB 3.9MB/s Installing collected packages: urllib3 Successfully installed urllib3-1.26.0 You are using pip version 8.1.1, however version 21.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. root@a317929cf5d7:/urllib3\_test# pip3 freeze requirements.txt urllib3==1.26.0 You are using pip version 8.1.1, however version 21.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
Now that we have updated our container, it is time to commit it and create our image by running the following Docker command from the docker host.
docker commit ubuntu\_xenial PS C:\\Users\\docker> docker commit ubuntu\_xenial sha256:130de6a9e3822dd3bbf19131469f3de441967756cd6faf70f52c92183ddea37e
Let’s check that Docker created the image by running the docker images command.
docker images
As we can see in the text output below, Docker created the image.
PS C:\\Users\\docker> docker images REPOSITORY TAG IMAGE ID CREATED SIZE none none 130de6a9e382 About a minute ago 465MB ubuntu xenial 9ff95a467e45 12 days ago 135MB
However, to make our lives easier, let’s give the new image a tag to identify it by running the following command.
docker tag 130de6a9e382 our\_ubuntu\_image
If we rerun the docker images command, we can see that Docker updated the information.
PS C:\\Users\\docker> docker tag 130de6a9e382 our\_ubuntu\_image PS C:\\Users\\docker> docker images REPOSITORY TAG IMAGE ID CREATED SIZE our\_ubuntu\_image latest 130de6a9e382 3 minutes ago 465MB ubuntu xenial 9ff95a467e45 12 days ago 135MB
As Docker’s scanning feature only works on images and not containers, we are now ready to scan the image for any vulnerabilities. Let’s start the Docker container scanning by running the following command.
docker scan our\_ubuntu\_image
PS C:\\Users\\docker> docker scan our\_ubuntu\_image Docker Scan relies upon access to Snyk, a third party provider, do you consent to proceed using Snyk? (y/N)docker y / Analyzing container dependencies for our\_ubuntu\_image / Querying vulnerabilities database...
Once the scan completes, the final report indicates 193 issues, as shown in the text output below.
Package manager: deb Project name: docker-image|our\_ubuntu\_image Docker image: our\_ubuntu\_image Platform: linux/amd64 Licenses: enabled Tested 185 dependencies for known issues, found 193 issues.
Docker Container Scanning Results
If we study the vulnerability report, we find that Docker container scanning highlights some vulnerabilities that can be fixed, as shown in the text output below.
✗ Low severity vulnerability found in glibc/libc-bin Description: Improper Data Handling Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-345675 Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta From: glibc/libc-bin@2.23-0ubuntu11.2 From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2 Fixed in: 2.23-0ubuntu11.3 ✗ Low severity vulnerability found in glibc/libc-bin Description: Integer Underflow Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-571388 Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta From: glibc/libc-bin@2.23-0ubuntu11.2 From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2 Fixed in: 2.23-0ubuntu11.3
If we look at the message for the two vulnerabilities in the image, we need to update those particular packages to resolve the issue. First, we need to create a container from the new image to make sure we are working with the latest version. As we did previously, we need to execute the docker run command as shown in the image below.
docker run -it –name our\_ubuntu\_image our\_ubuntu\_image
PS C:\\Users\\docker> docker run -it --name our\_ubuntu\_image our\_ubuntu\_image root@0dfcbff2e3d1:/#
After executing the docker run command, we can run the following commands to update Ubuntu from the container’s command line.
apt update && apt upgrade root@0dfcbff2e3d1:/# **apt update && apt upgrade** Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease Reading package lists... Done Building dependency tree Reading state information... Done 4 packages can be upgraded. Run 'apt list --upgradable' to see them. Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: apt libapt-pkg5.0 libc-bin multiarch-support 4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 2458 kB of archives. After this operation, 70.7 kB of additional disk space will be used. Do you want to continue? \[Y/n\]
After running and installing the updates, we need to commit and create another Docker image to save the changes.
docker commit our\_ubuntu\_image **PS C:\\Users\\docker> docker commit our\_ubuntu\_image** sha256:bf5809ae94d91ea37ac94fcc7260d607d15925347267bba911e9d3e2308dd93f
We can check that Docker created the image successfully by running the docker images command.
docker images
As we can see in the text output below, Docker confirms the new updated image exists.
PS C:\\Users\\docker> docker images REPOSITORY TAG IMAGE ID CREATED SIZE none none bf5809ae94d9 2 minutes ago 476MB our\_ubuntu\_image latest 130de6a9e382 19 minutes ago 465MB ubuntu xenial 9ff95a467e45 12 days ago 135MB
As we did before, let’s give the new image a tag to identify it by running the following command.
docker tag_ _bf5809ae94d9_ _updated\_ubuntu\_image
If we rerun the docker images command, we can see that Docker updated the information.
PS C:\\Users\\docker> docker tag bf5809ae94d9 updated\_ubuntu\_image PS C:\\Users\\docker> **docker images** REPOSITORY TAG IMAGE ID CREATED SIZE updated\_ubuntu\_image latest bf5809ae94d9 3 minutes ago 476MB our\_ubuntu\_image latest 130de6a9e382 21 minutes ago 465MB ubuntu xenial 9ff95a467e45 12 days ago 135MB
We are now ready to scan the updated container to see if our efforts resolved any vulnerabilities. As before, to run docker scan execute the following command. Note that we are now scanning our updated image.
docker scan updated\_ubuntu\_image PS C:\\Users\\docker> **docker scan updated\_ubuntu\_image** / Analyzing container dependencies for updated\_ubuntu\_image / Querying vulnerabilities database...
Once the scan completes, the final report indicates 191 issues, as shown in the text output below. This result demonstrates that updating our Ubuntu container resolved two vulnerabilities.
Package manager: deb Project name: docker-image|updated\_ubuntu\_image Docker image: updated\_ubuntu\_image Platform: linux/amd64 Licenses: enabled Tested 185 dependencies for known issues, found 191 issues.
However, if we search both the original vulnerability report and the report generated after our updates, neither report contains an alert for CVE-2021-28363. If you recall, when we created our image, we installed a vulnerable version of urllib3. Since neither scan picked up this vulnerability, we need to consider alternatives that mitigate this risk.
Validating Docker container scanning with MergeBase
To test the validity of Docker scan’s analysis and reporting, let’s run the same scans using MergeBase. As in the first example, we first need to run a scan on the Docker image named our\_ubuntu\_image before we run any Ubuntu updates.
First, we need to get the docker image repository and tag by running the following command:
docker images
As we can see in the text output below, the repository name is ‘our_ubuntu_image,’ and its tag is ‘latest.’
REPOSITORY TAG IMAGE ID CREATED SIZE our\_ubuntu\_image latest 130de6a9e382 18 minutes ago 462MB ubuntu xenial 9ff95a467e45 3 weeks ago 135MB
Using the MergeBase command-line tool, we can scan this image using the following command:
java -jar mergebase.jar –mode=profile –name=our\_ubuntu\_image our\_ubuntu\_image:latest
If we look at the summary scan report on the MergeBase portal, we can see that it found 301 vulnerabilities. This number exceeds the 193 found by Docker scan by 108 vulnerabilities.
Below is the text output from the command-line tool. As you can see, it details multiple CVEs and categorizes them by severity.
java -jar mergebase.jar --mode=profile --name=updated\_ubuntu\_image updated\_ubuntu\_image:latest Saving application profile to https://demo.mergebase.com Vulnerable Files Overview ========================= CRITICAL (12) ----- CVE-2017-7614, CVE-2017-7226, CVE-2017-6969 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU) CVE-2017-8283 (dpkg-dev@1.18.4ubuntu1.7.UBUNTU) CVE-2017-8283 (dpkg@1.18.4ubuntu1.7.UBUNTU) CVE-2019-18218 (file@5.25-2ubuntu1.4.UBUNTU) CVE-2008-1530 (gnupg@1.4.20-1ubuntu3.3.UBUNTU) CVE-2016-6309 (openssl@1.0.2g-1ubuntu4.19.UBUNTU) CVE-2017-12814 (perl-base@5.22.1-9ubuntu0.9.UBUNTU) CVE-2017-12814 (perl@5.22.1-9ubuntu0.9.UBUNTU) CVE-2018-1126 (procps@3.3.10-4ubuntu2.5.UBUNTU) CVE-2020-15801, CVE-2017-1000158 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU) CVE-2020-15801, CVE-2017-1000158 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU) CVE-2018-20839, CVE-2018-15688, 2 more... (systemd@229-4ubuntu21.31.UBUNTU) EXTRA-HIGH (7) ----- CVE-2019-3462 (apt@1.2.35.UBUNTU) CVE-2016-7543 (bash@4.3-14ubuntu1.4.UBUNTU) CVE-2020-6096 (libc-bin@2.23-0ubuntu11.3.UBUNTU) CVE-2020-6096 (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU) CVE-2019-11922 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU) CVE-2017-17522 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU) CVE-2017-17522 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU) HIGH (18) ----- CVE-2018-6557 (base-files@9.4ubuntu4.13.UBUNTU) CVE-2019-18276, CVE-2017-5932, CVE-2016-0634 (bash@4.3-14ubuntu1.4.UBUNTU) CVE-2021-3549, CVE-2021-20294, 73 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU) CVE-2015-8865, CVE-2014-9653 (file@5.25-2ubuntu1.4.UBUNTU) CVE-2021-3345, CVE-2019-14855, 2 more... (gnupg@1.4.20-1ubuntu3.3.UBUNTU) CVE-2018-5733, CVE-2018-5732, CVE-2017-3144 (isc-dhcp-client@4.3.3-5ubuntu12.10.UBUNTU) CVE-2021-3326, CVE-2019-9192, 4 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU) CVE-2021-3326, CVE-2019-9192, 4 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU) CVE-2016-6305 (openssl@1.0.2g-1ubuntu4.19.UBUNTU) CVE-2018-6952, CVE-2018-6951, CVE-2018-20969 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU) CVE-2016-2381 (perl@5.22.1-9ubuntu0.9.UBUNTU) CVE-2018-1125, CVE-2018-1124, 2 more... (procps@3.3.10-4ubuntu2.5.UBUNTU) CVE-2019-20916 (python3-pip@8.1.1-2ubuntu0.6.UBUNTU) CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU) CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU) CVE-2020-1712, CVE-2019-3844, 8 more... (systemd@229-4ubuntu21.31.UBUNTU) CVE-2016-6321 (tar@1.28-2.1ubuntu0.2.UBUNTU) CVE-2018-7738 (util-linux@2.27.1-6ubuntu3.10.UBUNTU) MEDIUM (18) ----- CVE-2018-0501, CVE-2016-1252 (apt@1.2.35.UBUNTU) CVE-2016-9401 (bash@4.3-14ubuntu1.4.UBUNTU) CVE-2021-3487, CVE-2021-20284, 80 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU) CVE-2017-18018 (coreutils@8.25-2ubuntu3~16.04.UBUNTU) CVE-2017-1000249, CVE-2014-9621, CVE-2014-9620 (file@5.25-2ubuntu1.4.UBUNTU) CVE-2011-2207 (gnupg@1.4.20-1ubuntu3.3.UBUNTU) CVE-2019-20795 (iproute2@4.3.0-1ubuntu3.16.04.5.UBUNTU) CVE-2020-27618, CVE-2019-7309, 7 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU) CVE-2020-27618, CVE-2019-7309, 7 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU) CVE-2021-24032, CVE-2021-24031 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU) CVE-2018-0739, CVE-2016-6308, CVE-2016-6307 (openssl@1.0.2g-1ubuntu4.19.UBUNTU) CVE-2019-20633, CVE-2016-10713 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU) CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU) CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU) CVE-2020-13776, CVE-2020-13529, 3 more... (systemd@229-4ubuntu21.31.UBUNTU) CVE-2021-20193 (tar@1.28-2.1ubuntu0.2.UBUNTU) CVE-2016-5011 (util-linux@2.27.1-6ubuntu3.10.UBUNTU) CVE-2021-28363 requirements.txt (urllib3@1.26.0) LOW (4) ----- USN-3156-0002 (apt@1.2.35.UBUNTU) CVE-2020-35448 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU) CVE-2019-1552 (openssl@1.0.2g-1ubuntu4.19.UBUNTU) USN-4120-0002, CVE-2019-20386, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)
Included in the list of vulnerabilities identified by MergeBase is the vulnerable version of urllib3 we installed.
To ensure we are comparing like for like, we should also run the MergeBase scan on the updated image after applying the Ubuntu updates as we did with Docker scan. We can reuse the same MergeBase command line after updating the relevant parameters.
java -jar mergebase.jar –mode=profile –name=updated\_ubuntu\_image updated\_ubuntu\_image:latest
Interestingly, the results for the updated Ubuntu image are identical before and after running the updates.
We can deduct the following outcomes from this exercise:
- Docker Scan and MergeBase find different vulnerabilities on the same containers.
- MergeBase provides far more detailed results when it comes to the actual software components installed on a container. The text output from the command line tool and the report it generates even details the relevant CVEs.
- The Docker scan results change after running system updates while Mergebase remains the same. This difference indicates that Docker scan focuses more on the operating system while MergeBase concentrates on its specialty, Software Composition Analysis.
Docker Container Scanning Limitations
Our rudimentary experiment shows that default container scanning solutions do not find application-layer vulnerabilities. Consequently, even though these tools provide a critical service that helps you find security issues in your containerized application, they do not offer a complete solution.
For example, if we build a Python app and use the vulnerable urllib3 library, our containerized app is not secure. In that instance, a container scan will not show any vulnerabilities, but an attacker could create a rogue certificate authority and compromise your containerized service.
However, this simple illustration is only one example. If we consider that researchers found 18,325 vulnerabilities in 2020 and 7,545 in the first five months of 2021, protecting your containerized application requires multiple layers of security. If you only rely on container scanning solutions, you are not getting the visibility you need for application-layer vulnerabilities, as our urllib3 example demonstrates. What you need is a layered defense-in-depth security approach.
Defense in Depth for Containerized Applications
While containerized applications offer several benefits, such as application consistency, scalability, and cost-effectiveness, managing all the moving parts that provide the platform running your app can be challenging. Since hosts, images, and containers make up a standard Docker architecture, a vulnerability in any of these three critical components could lead to a security incident.
For example, should a threat actor manage to compromise an image on Docker Hub, it could lead to a devastating supply chain attack. A vulnerability in your Docker host, whether you use a cloud platform, Windows, or Linux server, could also lead to a compromise of your container. Finally, there is the container itself. As we have illustrated with our simple example, standard container scanning apps do not find application-layer vulnerabilities.
Implementing defense-in-depth security for your containerized application requires a holistic approach. First, you need to secure your platform from external threats by implementing the relevant network segmentation, firewalls, and similar technologies that prevent unauthorized access. Then you need to continuously harden your hosts, images, and containers by leveraging Docker container scanning and implementing patches and upgrades. The final element you need to secure in your defense-in-depth strategy is your application layer. Since firewalls and vulnerability scanners that focus on hosts, images, and containers do not identify any application-related security threats, you need a solution to identify and mitigate this risk. As our example illustrated, Software Composition Analysis (SCA) tools, like MergeBase, provide visibility into the real risk of enterprise applications.
Since most, if not all, modern applications leverage external libraries and packages, identifying vulnerabilities in these elements is vital in securing your application. If we revisit our urllib3 example, scanning the containerized application with an SCA tool, like MergeBase found the vulnerability, and 107 others the Docker scan failed to identify.
ps. Check also our blog post on how to scanning a firecracker microVM with MergeBase.
