Blog

Scanning a firecracker microVM with MergeBase

Firecracker is a virtual machine monitor that allows you to create and manage microVMs. It leverages the Linux Kernel-based Virtual Machine (KVM) and utilizes a minimalist design for increased security. As firecracker microVMs do not include unnecessary devices and guest functionality, they provide a reduced memory footprint and attack surface area. 

The firecracker architecture is used by and integrated with several infrastructure solutions such as appfleet, containerd, fly.io, and OpenNebula. In this article, we will be building a firecracker containerd microVM and scan it for any known vulnerabilities with MergeBase.

Building a firecracker microVM

As firecracker leverages KVM, you need to run it on a bare metal server that supports virtualization, such as an AWS i3.metal instance. You can run it on a virtual machine that utilizes nested virtualization. However, this platform is not supported. The following script tests your current environment and will let you know if it is capable of running firecracker.  

#!/bin/bash
err=""; \
[ "$(uname) $(uname -m)" = "Linux x86_64" ] \
 || err="ERROR: your system is not Linux x86_64."; \
[ -r /dev/kvm ] && [ -w /dev/kvm ] \
 || err="$err\nERROR: /dev/kvm is innaccessible."; \
(( $(uname -r | cut -d. -f1)*1000 + $(uname -r | cut -d. -f2) >= 4014 )) \
 || err="$err\nERROR: your kernel version ($(uname -r)) is too old."; \

dmesg | grep -i "hypervisor detected" \
 && echo "WARNING: you are running in a virtual machine. Firecracker is not well tested under nested virtualization."; \

[ -z "$err" ] && echo "Your system looks ready for Firecracker!" || echo -e "$err"

If your system can run firecracker, the script will confirm it, as illustrated in the output below.

root@firecracker:~# bash test.sh
[ 0.000000] Hypervisor detected: KVM
root@firecracker:~# 

Install the latest version of Go

The firecracker team have created a quick start guide you can use to get your firecracker-containerd instance up and running. However, if you run the script, it will fail as it uses Go modules that leverage later versions of the Go programming language. The workaround for this challenge is to install the latest version of Go manually, as illustrated in the steps below.

Download the latest version of Go

cd /tmp
wget https://golang.org/dl/go1.<Version_Number>.linux-amd64.tar.gz

Extract the tarball to the installation location – /usr/local

tar -C /usr/local -xzf go1.<Version_Number>.linux-amd64.tar.gz

Set the environment

echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.profile
echo "export GOPATH=~/.go" >> ~/.profile
source ~/.profile

 If you have installed the latest version of Go successfully, running the command go version will verify it, as shown in the output below.

root@firecracker:/tmp# go version
go version go1.16.5 linux/amd64
root@firecracker:/tmp#

Download and install the required dependencies on a Debian based instance

Now that you have the latest version of Go installed, you can run the script below provided by the firecracker team that downloads and installs the required dependencies on a Debian based instance.

#!/bin/bash
 
cd ~
 
# Install git, Go 1.13, make, curl
sudo mkdir -p /etc/apt/sources.list.d
echo "deb http://ftp.debian.org/debian buster-backports main" | \
  sudo tee /etc/apt/sources.list.d/buster-backports.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
  install --yes \
  golang-1.13 \
  make \
  git \
  curl \
  e2fsprogs \
  util-linux \
  bc \
  gnupg
 
# Debian's Go 1.13 package installs "go" command under /usr/lib/go-1.13/bin
export PATH=/usr/lib/go-1.13/bin:$PATH
 
cd ~
 
# Install Docker CE
# Docker CE includes containerd, but we need a separate containerd binary, built
# in a later step
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt-key finger docker@docker.com | grep '9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88' || echo '**Cannot find Docker key**'
echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
     install --yes \
     docker-ce aufs-tools-
sudo usermod -aG docker $(whoami)
 
# Install device-mapper
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y dmsetup

Download and install firecracker-containerd

Once you have downloaded and installed the dependencies, you can move on to the next step and get firecracker-containerd downloaded and installed. Again, we will use the script below provided by the firecracker team.

#!/bin/bash
cd ~
 
# Check out firecracker-containerd and build it. This includes:
# * firecracker-containerd runtime, a containerd v2 runtime
# * firecracker-containerd agent, an inside-VM component
# * runc, to run containers inside the VM
# * a Debian-based root filesystem configured as read-only with a read-write
# overlay
# * firecracker-containerd, an alternative containerd binary that includes the
# firecracker VM lifecycle plugin and API
# * tc-redirect-tap and other CNI dependencies that enable VMs to start with
# access to networks available on the host
git clone https://github.com/firecracker-microvm/firecracker-containerd.git
cd firecracker-containerd
sg docker -c 'make all image firecracker'
sudo make install install-firecracker demo-network
 
cd ~
 
# Download kernel
curl -fsSL -o hello-vmlinux.bin https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux.bin
 
# Configure our firecracker-containerd binary to use our new snapshotter and
# separate storage from the default containerd binary
sudo mkdir -p /etc/firecracker-containerd
sudo mkdir -p /var/lib/firecracker-containerd/containerd
# Create the shim base directory for which firecracker-containerd will run the
# shim from
sudo mkdir -p /var/lib/firecracker-containerd
sudo tee /etc/firecracker-containerd/config.toml <<EOF
disabled_plugins = ["cri"]
root = "/var/lib/firecracker-containerd/containerd"
state = "/run/firecracker-containerd"
[grpc]
  address = "/run/firecracker-containerd/containerd.sock"
[plugins]
  [plugins.devmapper]
    pool_name = "fc-dev-thinpool"
    base_image_size = "10GB"
    root_path = "/var/lib/firecracker-containerd/snapshotter/devmapper"
 
[debug]
  level = "debug"
EOF
 
# Setup device mapper thin pool
sudo mkdir -p /var/lib/firecracker-containerd/snapshotter/devmapper
cd /var/lib/firecracker-containerd/snapshotter/devmapper
DIR=/var/lib/firecracker-containerd/snapshotter/devmapper
POOL=fc-dev-thinpool
 
if [[ ! -f "${DIR}/data" ]]; then
    sudo touch "${DIR}/data"
    sudo truncate -s 100G "${DIR}/data"
fi
 
if [[ ! -f "${DIR}/metadata" ]]; then
    sudo touch "${DIR}/metadata"
    sudo truncate -s 2G "${DIR}/metadata"
fi
 
DATADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/data)"
if [[ -z "${DATADEV}" ]]; then
    DATADEV="$(sudo losetup --find --show ${DIR}/data)"
fi
 
METADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/metadata)"
if [[ -z "${METADEV}" ]]; then
    METADEV="$(sudo losetup --find --show ${DIR}/metadata)"
fi
 
SECTORSIZE=512
DATASIZE="$(sudo blockdev --getsize64 -q ${DATADEV})"
LENGTH_SECTORS=$(bc <<< "${DATASIZE}/${SECTORSIZE}")
DATA_BLOCK_SIZE=128
LOW_WATER_MARK=32768
THINP_TABLE="0 ${LENGTH_SECTORS} thin-pool ${METADEV} ${DATADEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK} 1 skip_block_zeroing"
echo "${THINP_TABLE}"
 
if ! $(sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
    sudo dmsetup create "${POOL}" --table "${THINP_TABLE}"
fi
 
cd ~
 
# Configure the aws.firecracker runtime
# The long kernel command-line configures systemd inside the Debian-based image
# and uses a special init process to create a read-write overlay on top of the
# read-only image.
sudo mkdir -p /var/lib/firecracker-containerd/runtime
sudo cp ~/firecracker-containerd/tools/image-builder/rootfs.img /var/lib/firecracker-containerd/runtime/default-rootfs.img
sudo cp ~/hello-vmlinux.bin /var/lib/firecracker-containerd/runtime/default-vmlinux.bin
sudo mkdir -p /etc/containerd
sudo tee /etc/containerd/firecracker-runtime.json <<EOF
{
  "firecracker_binary_path": "/usr/local/bin/firecracker",
  "cpu_template": "T2",
  "log_fifo": "fc-logs.fifo",
  "log_levels": ["debug"],
  "metrics_fifo": "fc-metrics.fifo",
  "kernel_args": "console=ttyS0 noapic reboot=k panic=1 pci=off nomodules ro systemd.journald.forward_to_console systemd.unit=firecracker.target init=/sbin/overlay-init",
  "default_network_interfaces": [{
    "CNIConfig": {
      "NetworkName": "fcnet",
      "InterfaceName": "veth0"
    }
  }]
}
EOF
 

Pull an image and run the container

After you have downloaded and installed firecracker-containerd, you can now pull an image and run the microVM container. The following script starts firecracker-containerd.

firecracker-containerd --config /etc/firecracker-containerd/config.toml

 If you are successful, the terminal will present you with an output similar to the text below.

root@firecracker:/tmp# firecracker-containerd --config /etc/firecracker-containerd/config.toml
INFO[2021-07-09T06:09:51.535598043Z] starting containerd revision=c2323bc71886b3abfefd9afa53244740f70db0b8 version=1.5.2+unknown
INFO[2021-07-09T06:09:51.724317096Z] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
INFO[2021-07-09T06:09:51.724659848Z] loading plugin "io.containerd.snapshotter.v1.devmapper"... type=io.containerd.snapshotter.v1
INFO[2021-07-09T06:09:51.724787229Z] initializing pool device "fc-dev-thinpool"  
INFO[2021-07-09T06:09:51.726961441Z] using dmsetup:
Library version: 1.02.155 (2018-12-18)
Driver version: 4.39.0 
INFO[2021-07-09T06:09:52.153939660Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"... 
 

Now that firecracker-containerd is running; you need to open a new terminal, pull an image, and run the container. The script below pulls and runs the latest Debian image.

firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
     image pull \
     --snapshotter devmapper \
     docker.io/library/debian:latest
 
firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
     run \
     --snapshotter devmapper \
     --runtime aws.firecracker \
     --rm --tty --net-host \
     docker.io/library/debian:latest \
     test

As you can see in the output below, we have successfully started and logged into the microVM.

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> image pull \
> --snapshotter devmapper \
> docker.io/library/debian:latest

docker.io/library/debian:latest: resolved |++++++++++++++++++++++++++++++++++++++| 
index-sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b: done |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:5625c115ad881f19967a9b66416f8d40710bb307ad607d037f8ad8289260f75f: done |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0bc3020d05f1e08b41f1c5d54650a157b1690cde7fedb1fafbc9cda70ee2ec5c: done |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7a4951775d157843b47250a2a5cc7b561d2abe0b29ae6f19737a04635302eacf: done |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.1 s total: 48.1 M (15.4 MiB/s)                    
unpacking linux/amd64 sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b...
done: 3.373106679s

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> run \
> --snapshotter devmapper \
> --runtime aws.firecracker \
> --rm --tty --net-host \
> docker.io/library/debian:latest \
> test
root@microvm:/#  

Scanning a firecracker microVM with MergeBase

Now that we have a working firecracker-containerd image, we can scan it for vulnerabilities with MergeBase. T

The procedure is similar to the one for scanning Docker containers. First, you need to download the mergbase.jar file from the MergeBase portal, as shown in the image below.

 Then run the command to scan the image as shown below.

java -jar mergebase.jar --mode=profile --name=firecrackervm debian:buster-slim

As you can see in the output below, MergeBase identified several CVEs

root@firecracker:~# java -jar mergebase.jar --mode=profile --name=firecrackervm debian:buster-slim
Saving application profile to https://demo.mergebase.com
 
 Vulnerable Files Overview
 =========================
 
 EXTRA-HIGH (1)
 -----
 CVE-2020-6096 (libc-bin@2.28-10.DEBIAN)
 
 HIGH (2)
 -----
 CVE-2021-3326, CVE-2019-9192, CVE-2018-20796 (libc-bin@2.28-10.DEBIAN)
 CVE-2019-9923 (tar@1.30+dfsg-6.DEBIAN)
 
 MEDIUM (3)
 -----
 CVE-2019-5188, CVE-2019-5094 (e2fsprogs@1.44.5-1+deb10u3.DEBIAN)
 CVE-2020-27618, CVE-2019-7309, CVE-2019-25013 (libc-bin@2.28-10.DEBIAN)
 CVE-2021-20193 (tar@1.30+dfsg-6.DEBIAN)
 
 CVEs Overview
 =========================
 EXTRA-HIGH (1)
 -----
 CVE-2020-6096
 
 HIGH (4)
 -----
 CVE-2021-3326, CVE-2019-9923, CVE-2019-9192, CVE-2018-20796
 
 MEDIUM (6)
 -----
 CVE-2021-20193, CVE-2020-27618, CVE-2019-7309, CVE-2019-5188, CVE-2019-5094, CVE-2019-25013
 
Processing time: 5892 ms
 
MergeBase CLT version=v3.0.2 (build-id: bc04183)

 If we review the vulnerabilities on the MergeBase portal, the primary issue lies with three Debian components, namely libc-bin, tar, and e2fsprogs.

 Conclusion

If we compare the base scans from a Docker image to a Firecracker microVM, the outputs clearly show that microVMs have far fewer vulnerabilities. The previous article illustrated that a base Docker image had 329 vulnerabilities, as shown in the image below.

 

If we compare the 11 vulnerabilities found on a microVM, the reduced memory footprint and attack surface area clearly provide a more secure computing environment.

Try it out

Firecracker will really enhance your security through isolation of VM’s and, as we have seen by scanning a firecracker microVM with MergeBase in this article, far fewer vulnerabilities. Try it out yourself by using Firecracker and scanning it with MergeBase.

Ubuntu Docker Container Scanning – Is it enough to protect your app?

In this post, we are going to look at Docker container scanning:

  1. First, we will build an Ubuntu Docker container, add some software to it, run a Docker scan, and record the vulnerabilities.
  2. Then, we will try to resolve some of those vulnerabilities, run another scan, and examine the results.
  3. Finally, we will determine whether container scanning is enough to protect your applications and data from compromise. 

Building the Docker Container

We will build our Ubuntu Docker container using the Ubuntu Xenial image. Since it is a few versions behind, it will hopefully contain a few vulnerabilities. We are also going to install Python, PIP3, and a vulnerable version of urllib3. If we search the CVE database, we find CVE-2021-28363. According to this CVE, the urllib3 library 1.26.x before 1.26.4 omits SSL certificate validation in some cases involving HTTPS to HTTPS proxies.

The first step in our process is to pull the Ubuntu Xenial Docker image from Docker Hub and run it as a container on our Docker host. The command below will get the image, start the container in interactive mode, and name it ubuntu_xenial. 

docker run -it –name ubuntu_xenial ubuntu:xenial

Once the process completes, you will be presented with the Ubuntu prompt, as shown in the text output below.

PS C:\Users\docker> docker run -it --name ubuntu_xenial ubuntu:xenial

Unable to find image 'ubuntu:xenial' locally

xenial: Pulling from library/ubuntu

80bce60046fa: Pull complete

55a738a15540: Pull complete

e19cf0706c62: Pull complete

de4cdd6c27d1: Pull complete

Digest: sha256:9775877f420d453ef790e6832d77630a49b32a92b7bedf330adf4d8669f6600e

Status: Downloaded newer image for ubuntu:xenial

root@a317929cf5d7:/#

The next step is to install Python and PIP by running the following command:

apt update && apt install python3-pip

root@a317929cf5d7:/# apt update && apt install python3-pip

Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]

Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]

Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]

Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [2051 kB]
.....
Fetched 19.4 MB in 42s (458 kB/s)

Reading package lists... Done

Building dependency tree

Reading state information... Done

5 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done
.....
The following packages will be upgraded:

libc6

1 upgraded, 85 newly installed, 0 to remove and 4 not upgraded.

Need to get 98.7 MB of archives.

After this operation, 289 MB of additional disk space will be used.

Do you want to continue? [Y/n]

 

After installing Python and PIP, we are ready to install urllib3. However, since the development team fixed the vulnerability in version 1.26.4, we need to install an earlier vulnerable version. For the purposes of this exercise, let’s install version 1.26.0 from requirements.txt. 

We can check that our requirements.txt file includes our vulnerable version by running the cat command.

root@a317929cf5d7:/# cat requirements.txt

urllib3==1.26.0

We can use PIP to install the  requirements.txt file.

pip3 install -r requirements.txt

root@a317929cf5d7:/# pip3 -r requirements.txt

Collecting urllib3==1.26.0

  Downloading https://files.pythonhosted.org/packages/7e/a7/746338eb8addda2e7662ee5e10a9f85150aba013cd610c9569c17146b914/urllib3-1.26.0-py2.py3-none-any.whl (136kB)

    100% |################################| 143kB 3.9MB/s

Installing collected packages: urllib3

Successfully installed urllib3-1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

root@a317929cf5d7:/urllib3_test# pip3 freeze requirements.txt

urllib3==1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

Now that we have updated our container, it is time to commit it and create our image by running the following Docker command from the docker host.

docker commit ubuntu_xenial

PS C:\Users\docker> docker commit ubuntu_xenial

sha256:130de6a9e3822dd3bbf19131469f3de441967756cd6faf70f52c92183ddea37e

Let’s check that Docker created the image by running the docker images command.

docker images

As we can see in the text output below, Docker created the image. 

PS C:\Users\docker> docker images

REPOSITORY   TAG      IMAGE ID      CREATED              SIZE

<none>      <none>    130de6a9e382   About a minute ago   465MB

ubuntu       xenial   9ff95a467e45   12 days ago          135MB

However, to make our lives easier, let’s give the new image a tag to identify it by running the following command.

docker tag 130de6a9e382 our_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag 130de6a9e382 our_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED        SIZE

our_ubuntu_image  latest   130de6a9e382   3 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago     135MB

As Docker’s scanning feature only works on images and not containers, we are now ready to scan the image for any vulnerabilities. Let’s start the Docker container scanning by running the following command.

docker scan our_ubuntu_image

PS C:\Users\docker> docker scan our_ubuntu_image

Docker Scan relies upon access to Snyk, a third party provider, do you consent to proceed using Snyk? (y/N)docker y

/ Analyzing container dependencies for our_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 193 issues, as shown in the text output below.

Package manager:  deb

Project name:     docker-image|our_ubuntu_image

Docker image:     our_ubuntu_image

Platform:         linux/amd64

Licenses:        enabled

Tested 185 dependencies for known issues, found 193 issues.

Vulnerability Analysis and Resolution

If we study the vulnerability report, we find that Docker container scanning highlights some vulnerabilities that can be fixed, as shown in the text output below.

✗  Low severity vulnerability found in glibc/libc-bin

  Description: Improper Data Handling

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-345675

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

✗ Low severity vulnerability found in glibc/libc-bin

  Description: Integer Underflow

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-571388

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

If we look at the message for the two vulnerabilities in the image, we need to update those particular packages to resolve the issue. First, we need to create a container from the new image to make sure we are working with the latest version. As we did previously, we need to execute the docker run command as shown in the image below.

docker run -it –name our_ubuntu_image our_ubuntu_image

PS C:\Users\docker> docker run -it --name our_ubuntu_image our_ubuntu_image

root@0dfcbff2e3d1:/#

After executing the docker run command, we can run the following commands to update Ubuntu from the container’s command line.

apt update && apt upgrade

root@0dfcbff2e3d1:/# apt update && apt upgrade

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease

Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease

Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease

Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease

Reading package lists... Done

Building dependency tree

Reading state information... Done

4 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

The following packages will be upgraded:

  apt libapt-pkg5.0 libc-bin multiarch-support

4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Need to get 2458 kB of archives.

After this operation, 70.7 kB of additional disk space will be used.

Do you want to continue? [Y/n]

After running and installing the updates, we need to commit and create another Docker image to save the changes. 

docker commit our_ubuntu_image

PS C:\Users\docker> docker commit our_ubuntu_image

sha256:bf5809ae94d91ea37ac94fcc7260d607d15925347267bba911e9d3e2308dd93f

We can check that Docker created the image successfully by running the docker images command.

docker images

As we can see in the text output below, Docker confirms the new updated image exists.

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED          SIZE

<none>             <none>    bf5809ae94d9   2 minutes ago    476MB

our_ubuntu_image   latest    130de6a9e382   19 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago      135MB

As we did before, let’s give the new image a tag to identify it by running the following command.

docker tag bf5809ae94d9 updated_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag bf5809ae94d9 updated_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY             TAG       IMAGE ID       CREATED          SIZE

updated_ubuntu_image   latest    bf5809ae94d9   3 minutes ago   476MB

our_ubuntu_image       latest    130de6a9e382   21 minutes ago 465MB

ubuntu                 xenial    9ff95a467e45   12 days ago       135MB

We are now ready to scan the updated container to see if our efforts resolved any vulnerabilities. As before, to run docker scan execute the following command. Note that we are now scanning our updated image.

docker scan updated_ubuntu_image

PS C:\Users\docker> docker scan updated_ubuntu_image

/ Analyzing container dependencies for updated_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 191 issues, as shown in the text output below. This result demonstrates that updating our Ubuntu container resolved two vulnerabilities. 

Package manager:    deb

Project name:      docker-image|updated_ubuntu_image

Docker image:      updated_ubuntu_image

Platform:          linux/amd64

Licenses:         enabled

Tested 185 dependencies for known issues, found 191 issues.

However, if we search both the original vulnerability report and the report generated after our updates, neither report contains an alert for CVE-2021-28363. If you recall, when we created our image, we installed a vulnerable version of urllib3. Since neither scan picked up this vulnerability, we need to consider alternatives that mitigate this risk.

MergeBase Vulnerability Analysis and Resolution

To test the validity of Docker scan’s analysis and reporting, let’s run the same scans using MergeBase.

As in the first example, we first need to run a scan on the Docker image named our_ubuntu_image before we run any Ubuntu updates. 

First, we need to get the docker image repository and tag by running the following command:

docker images

As we can see in the text output below, the repository name is ‘our_ubuntu_image,’ and its tag is ‘latest.’

REPOSITORY TAG IMAGE ID CREATED SIZE

our_ubuntu_image latest 130de6a9e382 18 minutes ago 462MB

ubuntu xenial 9ff95a467e45 3 weeks ago 135MB

Using the MergeBase command-line tool, we can scan this image using the following command:

java -jar mergebase.jar –mode=profile –name=our_ubuntu_image our_ubuntu_image:latest

If we look at the summary scan report on the MergeBase portal, we can see that it found 301 vulnerabilities. This number exceeds the 193 found by Docker scan by 108 vulnerabilities.

Below is the text output from the command-line tool. As you can see, it details multiple CVEs and categorizes them by severity.

java -jar mergebase.jar --mode=profile --name=updated_ubuntu_image updated_ubuntu_image:latest

Saving application profile to https://demo.mergebase.com

  Vulnerable Files Overview

  =========================

  CRITICAL (12)

  -----

  CVE-2017-7614, CVE-2017-7226, CVE-2017-6969 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-8283 (dpkg-dev@1.18.4ubuntu1.7.UBUNTU)

  CVE-2017-8283 (dpkg@1.18.4ubuntu1.7.UBUNTU)

  CVE-2019-18218 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2008-1530 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2016-6309 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2017-12814 (perl-base@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2017-12814 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1126 (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2018-20839, CVE-2018-15688, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

  EXTRA-HIGH (7)

  -----

  CVE-2019-3462 (apt@1.2.35.UBUNTU)

  CVE-2016-7543 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2020-6096 (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-6096 (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2019-11922 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2017-17522 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2017-17522 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  HIGH (18)

  -----

  CVE-2018-6557 (base-files@9.4ubuntu4.13.UBUNTU)

  CVE-2019-18276, CVE-2017-5932, CVE-2016-0634 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3549, CVE-2021-20294, 73 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2015-8865, CVE-2014-9653 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2021-3345, CVE-2019-14855, 2 more... (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2018-5733, CVE-2018-5732, CVE-2017-3144 (isc-dhcp-client@4.3.3-5ubuntu12.10.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2016-6305 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2018-6952, CVE-2018-6951, CVE-2018-20969 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2016-2381 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1125, CVE-2018-1124, 2 more... (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2019-20916 (python3-pip@8.1.1-2ubuntu0.6.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-1712, CVE-2019-3844, 8 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2016-6321 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2018-7738 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  MEDIUM (18)

  -----

  CVE-2018-0501, CVE-2016-1252 (apt@1.2.35.UBUNTU)

  CVE-2016-9401 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3487, CVE-2021-20284, 80 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-18018 (coreutils@8.25-2ubuntu3~16.04.UBUNTU)

  CVE-2017-1000249, CVE-2014-9621, CVE-2014-9620 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2011-2207 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2019-20795 (iproute2@4.3.0-1ubuntu3.16.04.5.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-24032, CVE-2021-24031 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2018-0739, CVE-2016-6308, CVE-2016-6307 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2019-20633, CVE-2016-10713 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-13776, CVE-2020-13529, 3 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2021-20193 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2016-5011 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  CVE-2021-28363 requirements.txt (urllib3@1.26.0)

  LOW (4)

  -----

  USN-3156-0002 (apt@1.2.35.UBUNTU)

  CVE-2020-35448 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2019-1552 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  USN-4120-0002, CVE-2019-20386, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

Included in the list of vulnerabilities identified by MergeBase is the vulnerable version of urllib3 we installed.

To ensure we are comparing like for like, we should also run the MergeBase scan on the updated image after applying the Ubuntu updates as we did with Docker scan. We can reuse the same MergeBase command line after updating the relevant parameters.

java -jar mergebase.jar –mode=profile –name=updated_ubuntu_image updated_ubuntu_image:latest

Interestingly, the results for the updated Ubuntu image are identical before and after running the updates. 

We can deduct the following outcomes from this exercise:

  1. Docker Scan and MergeBase find different vulnerabilities on the same containers.
  2. MergeBase provides far more detailed results when it comes to the actual software components installed on a container. The text output from the command line tool and the report it generates even details the relevant CVEs. 
  3. The Docker scan results change after running system updates while Mergebase remains the same. This difference indicates that Docker scan focuses more on the operating system while MergeBase concentrates on its specialty, Software Composition Analysis.

Container Scanning Limitations

Our rudimentary experiment shows that default container scanning solutions do not find application-layer vulnerabilities. Consequently, even though these tools provide a critical service that helps you find security issues in your containerized application, they do not offer a complete solution. For example, if we build a Python app and use the vulnerable urllib3 library, our containerized app is not secure. In that instance, a container scan will not show any vulnerabilities, but an attacker could create a rogue certificate authority and compromise your containerized service.

However, this simple illustration is only one example. If we consider that researchers found 18,325 vulnerabilities in 2020 and 7,545 in the first five months of 2021, protecting your containerized application requires multiple layers of security. If you only rely on container scanning solutions, you are not getting the visibility you need for application-layer vulnerabilities, as our urllib3 example demonstrates. What you need is a layered defense-in-depth security approach. 

Defense in Depth for Containerized Applications

While containerized applications offer several benefits such as application consistency, scalability, and cost-effectiveness, managing all the moving parts that provide the platform running your app can be challenging. Since hosts, images, and containers make up a standard Docker architecture, a vulnerability in any of these three critical components could lead to a security incident. 

For example, should a threat actor manage to compromise an image on Docker Hub, it could lead to a devastating supply chain attack. A vulnerability in your Docker host, whether you use a cloud platform, Windows, or Linux server, could also lead to a compromise of your container. Finally, there is the container itself. As we have illustrated with our simple example, standard container scanning apps do not find application-layer vulnerabilities. 

Implementing defense-in-depth security for your containerized application requires a holistic approach. First, you need to secure your platform from external threats by implementing the relevant network segmentation, firewalls, and similar technologies that prevent unauthorized access. Then you need to continuously harden your hosts, images, and containers by leveraging Docker container scanning and implementing patches and upgrades. 

The final element you need to secure in your defense-in-depth strategy is your application layer. Since firewalls, and vulnerability scanners that focus on hosts, images, and containers, do not identify any application-related security threats, you need a solution to identify and mitigate this risk. As our example illustrated, Software Composition Analysis (SCA) tools like MergeBase, provide visibility into the real risk of enterprise applications. Since most, if not all, modern applications leverage external libraries and packages, identifying vulnerabilities in these elements is vital in securing your application. If we revisit our urllib3 example, scanning the containerized application with an SCA tool like MergeBase found the vulnerability and 107 others the Docker scan failed to identify.

Container Security Tools Comparison for Vulnerability Scans

Recently, we took on a new challenge: compare 5 popular container security tools, including our solution. We wanted to see how the products stack up against each other. How did they do? Read on to find out!

TL; DR:

Containers have been causing waves in IT and dev circles since 2013 when Docker’s container technology was launched. They have revolutionized deployment adding both speed and stability and have become critical for most IT operations, so securing them is a priority for all of us. How well do various tools do that? See the table below:

ToolStep 1 (Squid)Step 2 (Patched)Step 3 (Add App)Result
MergeBase312ALL
Aqua000NONE
Snyk2002
Docker Hub1001
Quay000NONE

But before we get into the details, it’s worthwhile to quickly revisit the importance of application container security in the modern-day development landscape.

The Importance of Container Security

Containers enable developers to run applications quickly and reliably when moved from one computing environment to another. But despite their many advantages – including increased application isolation – containers also amplify security risks. Increasing adoption in production environments makes them attractive to malicious actors. Since traditional network security solutions cannot always protect against lateral attacks, a lot of effort goes into developing application container security solutions.

Container security refers to the tools (e.g. Docker container security solutions) and policies implemented to protect container integrity and reliability, mitigate risk, and minimize vulnerabilities.

Container Security Tools Compared

To protect containers from attacks, many security tools are available. Usually, they audit the Common Vulnerabilities and Exposures (CVE) set by the National Vulnerability Database (NVD), or the benchmarks set by the Center for Internet Security (CIS).
Most containerized applications and their underlying infrastructure are distributed widely and highly dynamic. In this scenario, manual vulnerability scanning can be time-consuming and resource-intensive. To reduce operational overhead, many tools offer automation. Some focus on specific aspects of the cloud-native ecosystem, e.g. runtime security.
For our analysis, we picked 5 popular automated container scanners:

  • Aqua
  • Snyk
  • Docker Hub
  • Quay
  • MergeBase

Methodology

Before starting our analysis, we set up three images. The images are a logical progression where start with a vulnerable version of squid, then patch it and then add a vulnerable proprietary library, a proxy for applications you might produce and deploy in Docker images

container scanning expectations
  1. Seeded with a vulnerable version of Squid, a caching and forwarding HTTP web proxy
  2. Patch Squid to latest safe version
  3. Download a proprietary jar file that is vulnerable. Your own applications would typically fall in this category and it is challenging for most container scanning tools to analyze these.

We expected that each application would find vulnerabilities in all these steps.

However, this is not quite what happened!

Before we reveal the results of our tool comparison, here’s a sequence of steps that shows the Docker files used to build the images. Also, for readers planning to replicate our experiment, bear in mind that vulnerability scanning is sensitive to the date of the scan. We completed this experiment in early April 2021. New vulnerabilities may have been found and published since then, and security scanning tools themselves may have also changed.

Procedure to Build Images with Docker Files

If you need the build scripts, please ask us. We believe in transparency and are happy to provide them.

Results of Container Scanner Tool Comparison & Analysis

1. Aqua

For teams wondering how to secure Docker containers, Aqua claims to provide “enterprise-grade security for Docker environments” from development to production. Its tool scans images for vulnerabilities, malware, configuration issues, etc. for continuous image assurance. Its vulnerabilities database is aggregated from multiple, constantly-updated data streams to increase detection accuracy and provide better protection.

Aqua container scanning did not find any vulnerabilities

Despite these claims, the tool didn’t quite make the cut in our test. In fact, Aqua found no vulnerabilities at all, raising doubts about its effectiveness.

2. Snyk

Snyk helps teams automatically find, prioritize and fix vulnerabilities in containers throughout the container lifecycle. It can detect vulnerable dependencies during coding, prevent new vulnerabilities from passing through the build process, and test the production environment for newly-disclosed vulnerabilities.

Snyk says that it has fixed over 5 million container vulnerabilities. But during our tests, Snyk found two vulnerabilities in Step 1:

  • CVE-2020-25097 (Squid)
  • CVE-2021-30139 (apk-tools)

Snyk did not find any vulnerabilities from Steps 2 and 3.

3. Docker Hub

Docker hub container scanning did find only one vulnerability

When a Docker image is pushed to Docker Hub, it automatically scans it for vulnerabilities. Teams can review the security state of images, and fix identified issues for more secure deployments. The vulnerability report displays vulnerabilities, and sorts them according to severity. It also displays information about the:

  • Package containing the vulnerability
  • Version in which it was introduced
  • Whether the vulnerability is fixed in a later version

In our analysis, we found that this Docker container security scanner is not effective at finding all vulnerabilities. During testing, it only found one vulnerability from Step 1.

4. Quay

Quay container scanning did not find any vulnerabilities

Quay automatically scans containers to provide a real-time view of known vulnerabilities. The scan report displays vulnerabilities by severity level: Low, Medium and High. It also specifies whether patches are available.

But in our vulnerability test, Quay found no vulnerabilities. For all 3 steps, the report displayed a “passed” status for the security scan.

And now, we come to the final tool in our analysis: our own MergeBase tool.

5. MergeBase

In our analysis, only MergeBase found all vulnerabilities, including those the other tools missed:

MergeBase container scanning finds all vulnerabilities
  • CVE-2021-28116 (Squid) for which no patch is available
  • CVE-2016-5725 in the application, a directory traversal vulnerability in JCraft JSch before 0.1.54 on Windows, when the mode is ChannelSftp (source: CVE Mitre)

In summary, MergeBase found:

  • 3 vulnerabilities in Step 1
  • 1 vulnerability in Step 2
  • 2 vulnerabilities in Step 3

Comparison Conclusion

In containerized environments, the deployment pipeline is often standardized across different dev teams. Container scanning can help find vulnerabilities and take proactive action to fix security gaps. Securing containers and building security into the CI/CD their pipeline can help reduce the size of the attack surface.

However, different container scanning solutions yield inconsistent results on the same environment. Worse, many solutions fall short of their claims to help strengthen end-to-end container security.

In our analysis of 5 application container security tools, we found that our tool MergeBase was the only one that could find all vulnerabilities in our testing environment. Thus, compared to other tools, MergeBase provides complete DevSecOps coverage and reliable container security.

Want to know more about MergeBase? Take a look here!

Discover More from MergeBase

Core Product

BuildGreen is a powerful solution for identifying the real risk of open source at build time or in existing applications

Learn how BuildGreen can protects your Enterprise

Add RunTime Protection

RunGreen detects and defends against known-vulnerabilities at runtime.

Learn why Runtime Protection Matters

Optional Developer Add-on

CodeGreen is an early-warning defence for your in-house development and integrates directly into code repositories

Quick Start - For Free