Blog

Scanning a firecracker microVM with MergeBase

Firecracker is a virtual machine monitor that allows you to create and manage microVMs. It leverages the Linux Kernel-based Virtual Machine (KVM) and utilizes a minimalist design for increased security. As firecracker microVMs do not include unnecessary devices and guest functionality, they provide a reduced memory footprint and attack surface area. 

The firecracker architecture is used by and integrated with several infrastructure solutions such as appfleet, containerd, fly.io, and OpenNebula. In this article, we will be building a firecracker containerd microVM and scan it for any known vulnerabilities with MergeBase.

Building a firecracker microVM

As firecracker leverages KVM, you need to run it on a bare metal server that supports virtualization, such as an AWS i3.metal instance. You can run it on a virtual machine that utilizes nested virtualization. However, this platform is not supported. The following script tests your current environment and will let you know if it is capable of running firecracker.  

#!/bin/bash
err=""; \
[ "$(uname) $(uname -m)" = "Linux x86_64" ] \
 || err="ERROR: your system is not Linux x86_64."; \
[ -r /dev/kvm ] && [ -w /dev/kvm ] \
 || err="$err\nERROR: /dev/kvm is innaccessible."; \
(( $(uname -r | cut -d. -f1)*1000 + $(uname -r | cut -d. -f2) >= 4014 )) \
 || err="$err\nERROR: your kernel version ($(uname -r)) is too old."; \

dmesg | grep -i "hypervisor detected" \
 && echo "WARNING: you are running in a virtual machine. Firecracker is not well tested under nested virtualization."; \

[ -z "$err" ] && echo "Your system looks ready for Firecracker!" || echo -e "$err"

If your system can run firecracker, the script will confirm it, as illustrated in the output below.

root@firecracker:~# bash test.sh
[ 0.000000] Hypervisor detected: KVM
root@firecracker:~# 

Install the latest version of Go

The firecracker team have created a quick start guide you can use to get your firecracker-containerd instance up and running. However, if you run the script, it will fail as it uses Go modules that leverage later versions of the Go programming language. The workaround for this challenge is to install the latest version of Go manually, as illustrated in the steps below.

Download the latest version of Go

cd /tmp
wget https://golang.org/dl/go1.<Version_Number>.linux-amd64.tar.gz

Extract the tarball to the installation location – /usr/local

tar -C /usr/local -xzf go1.<Version_Number>.linux-amd64.tar.gz

Set the environment

echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.profile
echo "export GOPATH=~/.go" >> ~/.profile
source ~/.profile

 If you have installed the latest version of Go successfully, running the command go version will verify it, as shown in the output below.

root@firecracker:/tmp# go version
go version go1.16.5 linux/amd64
root@firecracker:/tmp#

Download and install the required dependencies on a Debian based instance

Now that you have the latest version of Go installed, you can run the script below provided by the firecracker team that downloads and installs the required dependencies on a Debian based instance.

#!/bin/bash
 
cd ~
 
# Install git, Go 1.13, make, curl
sudo mkdir -p /etc/apt/sources.list.d
echo "deb http://ftp.debian.org/debian buster-backports main" | \
  sudo tee /etc/apt/sources.list.d/buster-backports.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
  install --yes \
  golang-1.13 \
  make \
  git \
  curl \
  e2fsprogs \
  util-linux \
  bc \
  gnupg
 
# Debian's Go 1.13 package installs "go" command under /usr/lib/go-1.13/bin
export PATH=/usr/lib/go-1.13/bin:$PATH
 
cd ~
 
# Install Docker CE
# Docker CE includes containerd, but we need a separate containerd binary, built
# in a later step
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt-key finger docker@docker.com | grep '9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88' || echo '**Cannot find Docker key**'
echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
     install --yes \
     docker-ce aufs-tools-
sudo usermod -aG docker $(whoami)
 
# Install device-mapper
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y dmsetup

Download and install firecracker-containerd

Once you have downloaded and installed the dependencies, you can move on to the next step and get firecracker-containerd downloaded and installed. Again, we will use the script below provided by the firecracker team.

#!/bin/bash
cd ~
 
# Check out firecracker-containerd and build it. This includes:
# * firecracker-containerd runtime, a containerd v2 runtime
# * firecracker-containerd agent, an inside-VM component
# * runc, to run containers inside the VM
# * a Debian-based root filesystem configured as read-only with a read-write
# overlay
# * firecracker-containerd, an alternative containerd binary that includes the
# firecracker VM lifecycle plugin and API
# * tc-redirect-tap and other CNI dependencies that enable VMs to start with
# access to networks available on the host
git clone https://github.com/firecracker-microvm/firecracker-containerd.git
cd firecracker-containerd
sg docker -c 'make all image firecracker'
sudo make install install-firecracker demo-network
 
cd ~
 
# Download kernel
curl -fsSL -o hello-vmlinux.bin https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux.bin
 
# Configure our firecracker-containerd binary to use our new snapshotter and
# separate storage from the default containerd binary
sudo mkdir -p /etc/firecracker-containerd
sudo mkdir -p /var/lib/firecracker-containerd/containerd
# Create the shim base directory for which firecracker-containerd will run the
# shim from
sudo mkdir -p /var/lib/firecracker-containerd
sudo tee /etc/firecracker-containerd/config.toml <<EOF
disabled_plugins = ["cri"]
root = "/var/lib/firecracker-containerd/containerd"
state = "/run/firecracker-containerd"
[grpc]
  address = "/run/firecracker-containerd/containerd.sock"
[plugins]
  [plugins.devmapper]
    pool_name = "fc-dev-thinpool"
    base_image_size = "10GB"
    root_path = "/var/lib/firecracker-containerd/snapshotter/devmapper"
 
[debug]
  level = "debug"
EOF
 
# Setup device mapper thin pool
sudo mkdir -p /var/lib/firecracker-containerd/snapshotter/devmapper
cd /var/lib/firecracker-containerd/snapshotter/devmapper
DIR=/var/lib/firecracker-containerd/snapshotter/devmapper
POOL=fc-dev-thinpool
 
if [[ ! -f "${DIR}/data" ]]; then
    sudo touch "${DIR}/data"
    sudo truncate -s 100G "${DIR}/data"
fi
 
if [[ ! -f "${DIR}/metadata" ]]; then
    sudo touch "${DIR}/metadata"
    sudo truncate -s 2G "${DIR}/metadata"
fi
 
DATADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/data)"
if [[ -z "${DATADEV}" ]]; then
    DATADEV="$(sudo losetup --find --show ${DIR}/data)"
fi
 
METADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/metadata)"
if [[ -z "${METADEV}" ]]; then
    METADEV="$(sudo losetup --find --show ${DIR}/metadata)"
fi
 
SECTORSIZE=512
DATASIZE="$(sudo blockdev --getsize64 -q ${DATADEV})"
LENGTH_SECTORS=$(bc <<< "${DATASIZE}/${SECTORSIZE}")
DATA_BLOCK_SIZE=128
LOW_WATER_MARK=32768
THINP_TABLE="0 ${LENGTH_SECTORS} thin-pool ${METADEV} ${DATADEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK} 1 skip_block_zeroing"
echo "${THINP_TABLE}"
 
if ! $(sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
    sudo dmsetup create "${POOL}" --table "${THINP_TABLE}"
fi
 
cd ~
 
# Configure the aws.firecracker runtime
# The long kernel command-line configures systemd inside the Debian-based image
# and uses a special init process to create a read-write overlay on top of the
# read-only image.
sudo mkdir -p /var/lib/firecracker-containerd/runtime
sudo cp ~/firecracker-containerd/tools/image-builder/rootfs.img /var/lib/firecracker-containerd/runtime/default-rootfs.img
sudo cp ~/hello-vmlinux.bin /var/lib/firecracker-containerd/runtime/default-vmlinux.bin
sudo mkdir -p /etc/containerd
sudo tee /etc/containerd/firecracker-runtime.json <<EOF
{
  "firecracker_binary_path": "/usr/local/bin/firecracker",
  "cpu_template": "T2",
  "log_fifo": "fc-logs.fifo",
  "log_levels": ["debug"],
  "metrics_fifo": "fc-metrics.fifo",
  "kernel_args": "console=ttyS0 noapic reboot=k panic=1 pci=off nomodules ro systemd.journald.forward_to_console systemd.unit=firecracker.target init=/sbin/overlay-init",
  "default_network_interfaces": [{
    "CNIConfig": {
      "NetworkName": "fcnet",
      "InterfaceName": "veth0"
    }
  }]
}
EOF
 

Pull an image and run the container

After you have downloaded and installed firecracker-containerd, you can now pull an image and run the microVM container. The following script starts firecracker-containerd.

firecracker-containerd --config /etc/firecracker-containerd/config.toml

 If you are successful, the terminal will present you with an output similar to the text below.

root@firecracker:/tmp# firecracker-containerd --config /etc/firecracker-containerd/config.toml
INFO[2021-07-09T06:09:51.535598043Z] starting containerd revision=c2323bc71886b3abfefd9afa53244740f70db0b8 version=1.5.2+unknown
INFO[2021-07-09T06:09:51.724317096Z] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
INFO[2021-07-09T06:09:51.724659848Z] loading plugin "io.containerd.snapshotter.v1.devmapper"... type=io.containerd.snapshotter.v1
INFO[2021-07-09T06:09:51.724787229Z] initializing pool device "fc-dev-thinpool"  
INFO[2021-07-09T06:09:51.726961441Z] using dmsetup:
Library version: 1.02.155 (2018-12-18)
Driver version: 4.39.0 
INFO[2021-07-09T06:09:52.153939660Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"... 
 

Now that firecracker-containerd is running; you need to open a new terminal, pull an image, and run the container. The script below pulls and runs the latest Debian image.

firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
     image pull \
     --snapshotter devmapper \
     docker.io/library/debian:latest
 
firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
     run \
     --snapshotter devmapper \
     --runtime aws.firecracker \
     --rm --tty --net-host \
     docker.io/library/debian:latest \
     test

As you can see in the output below, we have successfully started and logged into the microVM.

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> image pull \
> --snapshotter devmapper \
> docker.io/library/debian:latest

docker.io/library/debian:latest: resolved |++++++++++++++++++++++++++++++++++++++| 
index-sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b: done |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:5625c115ad881f19967a9b66416f8d40710bb307ad607d037f8ad8289260f75f: done |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0bc3020d05f1e08b41f1c5d54650a157b1690cde7fedb1fafbc9cda70ee2ec5c: done |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7a4951775d157843b47250a2a5cc7b561d2abe0b29ae6f19737a04635302eacf: done |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.1 s total: 48.1 M (15.4 MiB/s)                    
unpacking linux/amd64 sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b...
done: 3.373106679s

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> run \
> --snapshotter devmapper \
> --runtime aws.firecracker \
> --rm --tty --net-host \
> docker.io/library/debian:latest \
> test
root@microvm:/#  

Scanning a firecracker microVM with MergeBase

Now that we have a working firecracker-containerd image, we can scan it for vulnerabilities with MergeBase. T

The procedure is similar to the one for scanning Docker containers. First, you need to download the mergbase.jar file from the MergeBase portal, as shown in the image below.

 Then run the command to scan the image as shown below.

java -jar mergebase.jar --mode=profile --name=firecrackervm debian:buster-slim

As you can see in the output below, MergeBase identified several CVEs

root@firecracker:~# java -jar mergebase.jar --mode=profile --name=firecrackervm debian:buster-slim
Saving application profile to https://demo.mergebase.com
 
 Vulnerable Files Overview
 =========================
 
 EXTRA-HIGH (1)
 -----
 CVE-2020-6096 (libc-bin@2.28-10.DEBIAN)
 
 HIGH (2)
 -----
 CVE-2021-3326, CVE-2019-9192, CVE-2018-20796 (libc-bin@2.28-10.DEBIAN)
 CVE-2019-9923 (tar@1.30+dfsg-6.DEBIAN)
 
 MEDIUM (3)
 -----
 CVE-2019-5188, CVE-2019-5094 (e2fsprogs@1.44.5-1+deb10u3.DEBIAN)
 CVE-2020-27618, CVE-2019-7309, CVE-2019-25013 (libc-bin@2.28-10.DEBIAN)
 CVE-2021-20193 (tar@1.30+dfsg-6.DEBIAN)
 
 CVEs Overview
 =========================
 EXTRA-HIGH (1)
 -----
 CVE-2020-6096
 
 HIGH (4)
 -----
 CVE-2021-3326, CVE-2019-9923, CVE-2019-9192, CVE-2018-20796
 
 MEDIUM (6)
 -----
 CVE-2021-20193, CVE-2020-27618, CVE-2019-7309, CVE-2019-5188, CVE-2019-5094, CVE-2019-25013
 
Processing time: 5892 ms
 
MergeBase CLT version=v3.0.2 (build-id: bc04183)

 If we review the vulnerabilities on the MergeBase portal, the primary issue lies with three Debian components, namely libc-bin, tar, and e2fsprogs.

 Conclusion

If we compare the base scans from a Docker image to a Firecracker microVM, the outputs clearly show that microVMs have far fewer vulnerabilities. The previous article illustrated that a base Docker image had 329 vulnerabilities, as shown in the image below.

 

If we compare the 11 vulnerabilities found on a microVM, the reduced memory footprint and attack surface area clearly provide a more secure computing environment.

Try it out

Firecracker will really enhance your security through isolation of VM’s and, as we have seen by scanning a firecracker microVM with MergeBase in this article, far fewer vulnerabilities. Try it out yourself by using Firecracker and scanning it with MergeBase.

Ubuntu Docker Container Scanning – Is it enough to protect your app?

In this post, we are going to look at Docker container scanning:

  1. First, we will build an Ubuntu Docker container, add some software to it, run a Docker scan, and record the vulnerabilities.
  2. Then, we will try to resolve some of those vulnerabilities, run another scan, and examine the results.
  3. Finally, we will determine whether container scanning is enough to protect your applications and data from compromise. 

Building the Docker Container

We will build our Ubuntu Docker container using the Ubuntu Xenial image. Since it is a few versions behind, it will hopefully contain a few vulnerabilities. We are also going to install Python, PIP3, and a vulnerable version of urllib3. If we search the CVE database, we find CVE-2021-28363. According to this CVE, the urllib3 library 1.26.x before 1.26.4 omits SSL certificate validation in some cases involving HTTPS to HTTPS proxies.

The first step in our process is to pull the Ubuntu Xenial Docker image from Docker Hub and run it as a container on our Docker host. The command below will get the image, start the container in interactive mode, and name it ubuntu_xenial. 

docker run -it –name ubuntu_xenial ubuntu:xenial

Once the process completes, you will be presented with the Ubuntu prompt, as shown in the text output below.

PS C:\Users\docker> docker run -it --name ubuntu_xenial ubuntu:xenial

Unable to find image 'ubuntu:xenial' locally

xenial: Pulling from library/ubuntu

80bce60046fa: Pull complete

55a738a15540: Pull complete

e19cf0706c62: Pull complete

de4cdd6c27d1: Pull complete

Digest: sha256:9775877f420d453ef790e6832d77630a49b32a92b7bedf330adf4d8669f6600e

Status: Downloaded newer image for ubuntu:xenial

root@a317929cf5d7:/#

The next step is to install Python and PIP by running the following command:

apt update && apt install python3-pip

root@a317929cf5d7:/# apt update && apt install python3-pip

Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]

Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]

Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]

Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [2051 kB]
.....
Fetched 19.4 MB in 42s (458 kB/s)

Reading package lists... Done

Building dependency tree

Reading state information... Done

5 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done
.....
The following packages will be upgraded:

libc6

1 upgraded, 85 newly installed, 0 to remove and 4 not upgraded.

Need to get 98.7 MB of archives.

After this operation, 289 MB of additional disk space will be used.

Do you want to continue? [Y/n]

 

After installing Python and PIP, we are ready to install urllib3. However, since the development team fixed the vulnerability in version 1.26.4, we need to install an earlier vulnerable version. For the purposes of this exercise, let’s install version 1.26.0 from requirements.txt. 

We can check that our requirements.txt file includes our vulnerable version by running the cat command.

root@a317929cf5d7:/# cat requirements.txt

urllib3==1.26.0

We can use PIP to install the  requirements.txt file.

pip3 install -r requirements.txt

root@a317929cf5d7:/# pip3 -r requirements.txt

Collecting urllib3==1.26.0

  Downloading https://files.pythonhosted.org/packages/7e/a7/746338eb8addda2e7662ee5e10a9f85150aba013cd610c9569c17146b914/urllib3-1.26.0-py2.py3-none-any.whl (136kB)

    100% |################################| 143kB 3.9MB/s

Installing collected packages: urllib3

Successfully installed urllib3-1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

root@a317929cf5d7:/urllib3_test# pip3 freeze requirements.txt

urllib3==1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

Now that we have updated our container, it is time to commit it and create our image by running the following Docker command from the docker host.

docker commit ubuntu_xenial

PS C:\Users\docker> docker commit ubuntu_xenial

sha256:130de6a9e3822dd3bbf19131469f3de441967756cd6faf70f52c92183ddea37e

Let’s check that Docker created the image by running the docker images command.

docker images

As we can see in the text output below, Docker created the image. 

PS C:\Users\docker> docker images

REPOSITORY   TAG      IMAGE ID      CREATED              SIZE

<none>      <none>    130de6a9e382   About a minute ago   465MB

ubuntu       xenial   9ff95a467e45   12 days ago          135MB

However, to make our lives easier, let’s give the new image a tag to identify it by running the following command.

docker tag 130de6a9e382 our_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag 130de6a9e382 our_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED        SIZE

our_ubuntu_image  latest   130de6a9e382   3 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago     135MB

As Docker’s scanning feature only works on images and not containers, we are now ready to scan the image for any vulnerabilities. Let’s start the Docker container scanning by running the following command.

docker scan our_ubuntu_image

PS C:\Users\docker> docker scan our_ubuntu_image

Docker Scan relies upon access to Snyk, a third party provider, do you consent to proceed using Snyk? (y/N)docker y

/ Analyzing container dependencies for our_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 193 issues, as shown in the text output below.

Package manager:  deb

Project name:     docker-image|our_ubuntu_image

Docker image:     our_ubuntu_image

Platform:         linux/amd64

Licenses:        enabled

Tested 185 dependencies for known issues, found 193 issues.

Vulnerability Analysis and Resolution

If we study the vulnerability report, we find that Docker container scanning highlights some vulnerabilities that can be fixed, as shown in the text output below.

✗  Low severity vulnerability found in glibc/libc-bin

  Description: Improper Data Handling

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-345675

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

✗ Low severity vulnerability found in glibc/libc-bin

  Description: Integer Underflow

  Info: https://snyk.io/vuln/SNYK-UBUNTU1604-GLIBC-571388

  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

If we look at the message for the two vulnerabilities in the image, we need to update those particular packages to resolve the issue. First, we need to create a container from the new image to make sure we are working with the latest version. As we did previously, we need to execute the docker run command as shown in the image below.

docker run -it –name our_ubuntu_image our_ubuntu_image

PS C:\Users\docker> docker run -it --name our_ubuntu_image our_ubuntu_image

root@0dfcbff2e3d1:/#

After executing the docker run command, we can run the following commands to update Ubuntu from the container’s command line.

apt update && apt upgrade

root@0dfcbff2e3d1:/# apt update && apt upgrade

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease

Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease

Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease

Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease

Reading package lists... Done

Building dependency tree

Reading state information... Done

4 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

The following packages will be upgraded:

  apt libapt-pkg5.0 libc-bin multiarch-support

4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Need to get 2458 kB of archives.

After this operation, 70.7 kB of additional disk space will be used.

Do you want to continue? [Y/n]

After running and installing the updates, we need to commit and create another Docker image to save the changes. 

docker commit our_ubuntu_image

PS C:\Users\docker> docker commit our_ubuntu_image

sha256:bf5809ae94d91ea37ac94fcc7260d607d15925347267bba911e9d3e2308dd93f

We can check that Docker created the image successfully by running the docker images command.

docker images

As we can see in the text output below, Docker confirms the new updated image exists.

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED          SIZE

<none>             <none>    bf5809ae94d9   2 minutes ago    476MB

our_ubuntu_image   latest    130de6a9e382   19 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago      135MB

As we did before, let’s give the new image a tag to identify it by running the following command.

docker tag bf5809ae94d9 updated_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag bf5809ae94d9 updated_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY             TAG       IMAGE ID       CREATED          SIZE

updated_ubuntu_image   latest    bf5809ae94d9   3 minutes ago   476MB

our_ubuntu_image       latest    130de6a9e382   21 minutes ago 465MB

ubuntu                 xenial    9ff95a467e45   12 days ago       135MB

We are now ready to scan the updated container to see if our efforts resolved any vulnerabilities. As before, to run docker scan execute the following command. Note that we are now scanning our updated image.

docker scan updated_ubuntu_image

PS C:\Users\docker> docker scan updated_ubuntu_image

/ Analyzing container dependencies for updated_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 191 issues, as shown in the text output below. This result demonstrates that updating our Ubuntu container resolved two vulnerabilities. 

Package manager:    deb

Project name:      docker-image|updated_ubuntu_image

Docker image:      updated_ubuntu_image

Platform:          linux/amd64

Licenses:         enabled

Tested 185 dependencies for known issues, found 191 issues.

However, if we search both the original vulnerability report and the report generated after our updates, neither report contains an alert for CVE-2021-28363. If you recall, when we created our image, we installed a vulnerable version of urllib3. Since neither scan picked up this vulnerability, we need to consider alternatives that mitigate this risk.

MergeBase Vulnerability Analysis and Resolution

To test the validity of Docker scan’s analysis and reporting, let’s run the same scans using MergeBase.

As in the first example, we first need to run a scan on the Docker image named our_ubuntu_image before we run any Ubuntu updates. 

First, we need to get the docker image repository and tag by running the following command:

docker images

As we can see in the text output below, the repository name is ‘our_ubuntu_image,’ and its tag is ‘latest.’

REPOSITORY TAG IMAGE ID CREATED SIZE

our_ubuntu_image latest 130de6a9e382 18 minutes ago 462MB

ubuntu xenial 9ff95a467e45 3 weeks ago 135MB

Using the MergeBase command-line tool, we can scan this image using the following command:

java -jar mergebase.jar –mode=profile –name=our_ubuntu_image our_ubuntu_image:latest

If we look at the summary scan report on the MergeBase portal, we can see that it found 301 vulnerabilities. This number exceeds the 193 found by Docker scan by 108 vulnerabilities.

Below is the text output from the command-line tool. As you can see, it details multiple CVEs and categorizes them by severity.

java -jar mergebase.jar --mode=profile --name=updated_ubuntu_image updated_ubuntu_image:latest

Saving application profile to https://demo.mergebase.com

  Vulnerable Files Overview

  =========================

  CRITICAL (12)

  -----

  CVE-2017-7614, CVE-2017-7226, CVE-2017-6969 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-8283 (dpkg-dev@1.18.4ubuntu1.7.UBUNTU)

  CVE-2017-8283 (dpkg@1.18.4ubuntu1.7.UBUNTU)

  CVE-2019-18218 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2008-1530 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2016-6309 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2017-12814 (perl-base@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2017-12814 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1126 (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2018-20839, CVE-2018-15688, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

  EXTRA-HIGH (7)

  -----

  CVE-2019-3462 (apt@1.2.35.UBUNTU)

  CVE-2016-7543 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2020-6096 (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-6096 (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2019-11922 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2017-17522 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2017-17522 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  HIGH (18)

  -----

  CVE-2018-6557 (base-files@9.4ubuntu4.13.UBUNTU)

  CVE-2019-18276, CVE-2017-5932, CVE-2016-0634 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3549, CVE-2021-20294, 73 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2015-8865, CVE-2014-9653 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2021-3345, CVE-2019-14855, 2 more... (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2018-5733, CVE-2018-5732, CVE-2017-3144 (isc-dhcp-client@4.3.3-5ubuntu12.10.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2016-6305 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2018-6952, CVE-2018-6951, CVE-2018-20969 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2016-2381 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1125, CVE-2018-1124, 2 more... (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2019-20916 (python3-pip@8.1.1-2ubuntu0.6.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-1712, CVE-2019-3844, 8 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2016-6321 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2018-7738 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  MEDIUM (18)

  -----

  CVE-2018-0501, CVE-2016-1252 (apt@1.2.35.UBUNTU)

  CVE-2016-9401 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3487, CVE-2021-20284, 80 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-18018 (coreutils@8.25-2ubuntu3~16.04.UBUNTU)

  CVE-2017-1000249, CVE-2014-9621, CVE-2014-9620 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2011-2207 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2019-20795 (iproute2@4.3.0-1ubuntu3.16.04.5.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-24032, CVE-2021-24031 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2018-0739, CVE-2016-6308, CVE-2016-6307 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2019-20633, CVE-2016-10713 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-13776, CVE-2020-13529, 3 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2021-20193 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2016-5011 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  CVE-2021-28363 requirements.txt (urllib3@1.26.0)

  LOW (4)

  -----

  USN-3156-0002 (apt@1.2.35.UBUNTU)

  CVE-2020-35448 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2019-1552 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  USN-4120-0002, CVE-2019-20386, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

Included in the list of vulnerabilities identified by MergeBase is the vulnerable version of urllib3 we installed.

To ensure we are comparing like for like, we should also run the MergeBase scan on the updated image after applying the Ubuntu updates as we did with Docker scan. We can reuse the same MergeBase command line after updating the relevant parameters.

java -jar mergebase.jar –mode=profile –name=updated_ubuntu_image updated_ubuntu_image:latest

Interestingly, the results for the updated Ubuntu image are identical before and after running the updates. 

We can deduct the following outcomes from this exercise:

  1. Docker Scan and MergeBase find different vulnerabilities on the same containers.
  2. MergeBase provides far more detailed results when it comes to the actual software components installed on a container. The text output from the command line tool and the report it generates even details the relevant CVEs. 
  3. The Docker scan results change after running system updates while Mergebase remains the same. This difference indicates that Docker scan focuses more on the operating system while MergeBase concentrates on its specialty, Software Composition Analysis.

Container Scanning Limitations

Our rudimentary experiment shows that default container scanning solutions do not find application-layer vulnerabilities. Consequently, even though these tools provide a critical service that helps you find security issues in your containerized application, they do not offer a complete solution. For example, if we build a Python app and use the vulnerable urllib3 library, our containerized app is not secure. In that instance, a container scan will not show any vulnerabilities, but an attacker could create a rogue certificate authority and compromise your containerized service.

However, this simple illustration is only one example. If we consider that researchers found 18,325 vulnerabilities in 2020 and 7,545 in the first five months of 2021, protecting your containerized application requires multiple layers of security. If you only rely on container scanning solutions, you are not getting the visibility you need for application-layer vulnerabilities, as our urllib3 example demonstrates. What you need is a layered defense-in-depth security approach. 

Defense in Depth for Containerized Applications

While containerized applications offer several benefits such as application consistency, scalability, and cost-effectiveness, managing all the moving parts that provide the platform running your app can be challenging. Since hosts, images, and containers make up a standard Docker architecture, a vulnerability in any of these three critical components could lead to a security incident. 

For example, should a threat actor manage to compromise an image on Docker Hub, it could lead to a devastating supply chain attack. A vulnerability in your Docker host, whether you use a cloud platform, Windows, or Linux server, could also lead to a compromise of your container. Finally, there is the container itself. As we have illustrated with our simple example, standard container scanning apps do not find application-layer vulnerabilities. 

Implementing defense-in-depth security for your containerized application requires a holistic approach. First, you need to secure your platform from external threats by implementing the relevant network segmentation, firewalls, and similar technologies that prevent unauthorized access. Then you need to continuously harden your hosts, images, and containers by leveraging Docker container scanning and implementing patches and upgrades. 

The final element you need to secure in your defense-in-depth strategy is your application layer. Since firewalls, and vulnerability scanners that focus on hosts, images, and containers, do not identify any application-related security threats, you need a solution to identify and mitigate this risk. As our example illustrated, Software Composition Analysis (SCA) tools like MergeBase, provide visibility into the real risk of enterprise applications. Since most, if not all, modern applications leverage external libraries and packages, identifying vulnerabilities in these elements is vital in securing your application. If we revisit our urllib3 example, scanning the containerized application with an SCA tool like MergeBase found the vulnerability and 107 others the Docker scan failed to identify.

Container Security Tools Comparison for Vulnerability Scans

Recently, we took on a new challenge: compare 5 popular container security tools, including our solution. We wanted to see how the products stack up against each other. How did they do? Read on to find out!

TL; DR:

Containers have been causing waves in IT and dev circles since 2013 when Docker’s container technology was launched. They have revolutionized deployment adding both speed and stability and have become critical for most IT operations, so securing them is a priority for all of us. How well do various tools do that? See the table below:

ToolStep 1 (Squid)Step 2 (Patched)Step 3 (Add App)Result
MergeBase312ALL
Aqua000NONE
Snyk2002
Docker Hub1001
Quay000NONE

But before we get into the details, it’s worthwhile to quickly revisit the importance of application container security in the modern-day development landscape.

The Importance of Container Security

Containers enable developers to run applications quickly and reliably when moved from one computing environment to another. But despite their many advantages – including increased application isolation – containers also amplify security risks. Increasing adoption in production environments makes them attractive to malicious actors. Since traditional network security solutions cannot always protect against lateral attacks, a lot of effort goes into developing application container security solutions.

Container security refers to the tools (e.g. Docker container security solutions) and policies implemented to protect container integrity and reliability, mitigate risk, and minimize vulnerabilities.

Container Security Tools Compared

To protect containers from attacks, many security tools are available. Usually, they audit the Common Vulnerabilities and Exposures (CVE) set by the National Vulnerability Database (NVD), or the benchmarks set by the Center for Internet Security (CIS).
Most containerized applications and their underlying infrastructure are distributed widely and highly dynamic. In this scenario, manual vulnerability scanning can be time-consuming and resource-intensive. To reduce operational overhead, many tools offer automation. Some focus on specific aspects of the cloud-native ecosystem, e.g. runtime security.
For our analysis, we picked 5 popular automated container scanners:

  • Aqua
  • Snyk
  • Docker Hub
  • Quay
  • MergeBase

Methodology

Before starting our analysis, we set up three images. The images are a logical progression where start with a vulnerable version of squid, then patch it and then add a vulnerable proprietary library, a proxy for applications you might produce and deploy in Docker images

container scanning expectations
  1. Seeded with a vulnerable version of Squid, a caching and forwarding HTTP web proxy
  2. Patch Squid to latest safe version
  3. Download a proprietary jar file that is vulnerable. Your own applications would typically fall in this category and it is challenging for most container scanning tools to analyze these.

We expected that each application would find vulnerabilities in all these steps.

However, this is not quite what happened!

Before we reveal the results of our tool comparison, here’s a sequence of steps that shows the Docker files used to build the images. Also, for readers planning to replicate our experiment, bear in mind that vulnerability scanning is sensitive to the date of the scan. We completed this experiment in early April 2021. New vulnerabilities may have been found and published since then, and security scanning tools themselves may have also changed.

Procedure to Build Images with Docker Files

If you need the build scripts, please ask us. We believe in transparency and are happy to provide them.

Results of Container Scanner Tool Comparison & Analysis

1. Aqua

For teams wondering how to secure Docker containers, Aqua claims to provide “enterprise-grade security for Docker environments” from development to production. Its tool scans images for vulnerabilities, malware, configuration issues, etc. for continuous image assurance. Its vulnerabilities database is aggregated from multiple, constantly-updated data streams to increase detection accuracy and provide better protection.

Aqua container scanning did not find any vulnerabilities

Despite these claims, the tool didn’t quite make the cut in our test. In fact, Aqua found no vulnerabilities at all, raising doubts about its effectiveness.

2. Snyk

Snyk helps teams automatically find, prioritize and fix vulnerabilities in containers throughout the container lifecycle. It can detect vulnerable dependencies during coding, prevent new vulnerabilities from passing through the build process, and test the production environment for newly-disclosed vulnerabilities.

Snyk says that it has fixed over 5 million container vulnerabilities. But during our tests, Snyk found two vulnerabilities in Step 1:

  • CVE-2020-25097 (Squid)
  • CVE-2021-30139 (apk-tools)

Snyk did not find any vulnerabilities from Steps 2 and 3.

3. Docker Hub

Docker hub container scanning did find only one vulnerability

When a Docker image is pushed to Docker Hub, it automatically scans it for vulnerabilities. Teams can review the security state of images, and fix identified issues for more secure deployments. The vulnerability report displays vulnerabilities, and sorts them according to severity. It also displays information about the:

  • Package containing the vulnerability
  • Version in which it was introduced
  • Whether the vulnerability is fixed in a later version

In our analysis, we found that this Docker container security scanner is not effective at finding all vulnerabilities. During testing, it only found one vulnerability from Step 1.

4. Quay

Quay container scanning did not find any vulnerabilities

Quay automatically scans containers to provide a real-time view of known vulnerabilities. The scan report displays vulnerabilities by severity level: Low, Medium and High. It also specifies whether patches are available.

But in our vulnerability test, Quay found no vulnerabilities. For all 3 steps, the report displayed a “passed” status for the security scan.

And now, we come to the final tool in our analysis: our own MergeBase tool.

5. MergeBase

In our analysis, only MergeBase found all vulnerabilities, including those the other tools missed:

MergeBase container scanning finds all vulnerabilities
  • CVE-2021-28116 (Squid) for which no patch is available
  • CVE-2016-5725 in the application, a directory traversal vulnerability in JCraft JSch before 0.1.54 on Windows, when the mode is ChannelSftp (source: CVE Mitre)

In summary, MergeBase found:

  • 3 vulnerabilities in Step 1
  • 1 vulnerability in Step 2
  • 2 vulnerabilities in Step 3

Comparison Conclusion

In containerized environments, the deployment pipeline is often standardized across different dev teams. Container scanning can help find vulnerabilities and take proactive action to fix security gaps. Securing containers and building security into the CI/CD their pipeline can help reduce the size of the attack surface.

However, different container scanning solutions yield inconsistent results on the same environment. Worse, many solutions fall short of their claims to help strengthen end-to-end container security.

In our analysis of 5 application container security tools, we found that our tool MergeBase was the only one that could find all vulnerabilities in our testing environment. Thus, compared to other tools, MergeBase provides complete DevSecOps coverage and reliable container security.

Want to know more about MergeBase? Take a look here!

Enhance software supply chain security says White House

Summary

The White House exec order to improve the software supply chain security
White House executive cyber security order

Software supply chain security is a common theme in recent attacks such as on the Colonial Pipeline and SolarWinds. In response, President Biden released an executive order to improve the nation’s cyber security. The order is a testimony of the importance of the digital ecosystem today to our society, economy and way of life. Cyber security is critical in protecting it and the President’s clear message is that we need to do more.

The government want to improve its own cyber security practices and related agencies. It also proposes to remove barriers for information sharing, and enhancing software supply chain security. The software supply chain security is the only specific attack vector that is highlighted.

These attacks cause the most damage and at the same time businesses and governments are ill prepared. So what is a supply chain attack and how can it be mitigated?

What is a Software Supply Chain Attack?

A supply chain attack leverages the access of an external partner or provider to gain unauthorized entry to a system or network. It takes advantage of the inherent trust a target has in its suppliers, using it to infiltrate and launch the cyber-attack. Its use of stealth and its indirect ‘trusted’ approach make a supply chain attack an effective weapon in any threat actor’s arsenal.

Every organization and individual relies on third-party software in some way or another. When we install software, hardware, or use code from a trusted source, it is natural to assume that it hides no malicious intent. In addition to this inherent trust, a supply chain attack also uses the human element to bypass any perimeter security. As administrative users and software developers install software, hardware, or reuse third-party code, they do so on the internal network, bypassing any security controls that prevent external threats.

Supply chain attacks that target technology infrastructure come in many variations. Threat actors can infiltrate a software provider and embed malicious code that infects end users when they install or access the product. Another effective supply chain attack technique is infecting software code repositories that software developers leverage to create systems. Finally, threat actors can also infiltrate and infect the embedded software that operates the hardware on networking equipment, servers, and end-user devices.

What is a software supply chain?

Modern software development processes rely on code reuse to build systems rapidly and cost-effectively. By leveraging existing code, developers can quickly assemble a system with its needed components instead of coding the entire solution from a blank canvas. Typically, programmers either reuse internally developed software code or leverage third-party libraries and frameworks. These components and their dependencies form part of the software supply chain. In other words, a software supply chain is a list of elements that goes into or affects the code from development to production.

Almost every software application or service we use today leverages a software supply chain. For example, Netflix and Uber use Node.js, an open-source, server-side JavaScript platform that is well suited for scalable applications. Another example is WordPress, a content management system used to run nearly 40% of the world’s websites. The list goes on, but it suffices to say that organizations leverage software supply chains everywhere. Besides leveraging the frameworks and platforms mentioned, software developers also use code libraries to build their solutions. Services like GitHub and StackOverflow are valuable resources where developers can find libraries, code snippets, and advice to help them create solutions.

However, a software supply chain does not only pertain to software development. It can also refer to instances where organizations install and run third-party applications in their technology environment. For example, every organization leverages third-party software for email. It would be both inefficient and expensive to develop an in-house solution for this utility. The same goes for system monitoring, file sharing, security, and other commodities in a technology environment. All these third-party applications and the external code it uses in its custom-developed applications form part of an organization’s software supply chain.

The anatomy of a supply chain attack

A supply chain attack infects the third-party technologies organizations use. It then leverages this unauthorized access to infiltrate and attack their primary targets. Typically, supply chain attacks start when threat actors exploit a vulnerability to gain access to a supplier’s systems. Once they have gained entry, they embed malicious code into the supplier’s software or hardware with a particular payload. The threat actor then waits until the target organization or user runs the supplier’s infected software or installs its infected hardware. As this infiltration technique circumvents any perimeter security, its indirect attack methodology is highly effective. It is also successful in gaining access to secure environments as these attacks typically target less secure elements in the supply chain.

Supply chain attacks are not a new type of threat, but recent cases have raised their prominence in the public domain. If we look at past instances, the Target data breach where malware infected their Point of Sale systems occurred in 2013. In that instance, the attackers compromised the organization’s third-party refrigerator vendor to infect Target’s POS environment with malware that stole credit card details. Another significant example is the famous Stuxnet malware that nation-states used to sabotage Iran’s nuclear centrifuges in 2010. In this example, the attackers used the digital certificates of Realtek Semiconductor to make their malware look legitimate to system administrators and evade anti-virus.

More recently, in the SolarWinds supply chain attack, threat actors deployed malware during a routine update that emanated from SolarWinds’ servers. Every organization that ran the update was subsequently compromised, including technology companies and secure government agencies. As a result of this attack, the United States sanctioned Russia, believing that the Kremlin played a role in this mass infiltration. Other recent supply chain attacks include the narrowly averted PHP backdoor and the Code Dev incident.

These supply chain attack examples show that this technique has successfully infected many organizations around the world. What is of particular concern is that these supply chain attacks also succeeded in highly secure environments. The examples also show the extensive ramifications of a successful attack. One undetected infection can affect thousands of users and organizations.

Software supply chain security, the open-source risk

Many organizations use open-source software in some way, shape, or form. With open-source code present in 90% of modern applications, this vital element in the software development ecosystem is vulnerable to a supply chain attack. The recent PHP case mentioned earlier is a prime example. Modern software applications reuse open-source libraries, frameworks, and code snippets. Threat actors target these components as they are typically less secure.

The 2020 Sonatype State of the Software Supply Chain Report stated next-generation attacks increased by 430% in the preceding 12 months. Unlike commercial software, open-source relies on the community to ensure its security. However, it is up to the organizations that use the software to conduct regular analysis, security audits, and penetration tests.

Technology supply chain risk

The technology supply chain includes hardware and software. Although the focus of this article has been on software supply chain attacks, organizations cannot ignore the hardware risk. Numerous examples of mobile devices arriving with embedded malware and compromised networking equipment used to breach secure networks highlight this threat.

These instances illustrate that business and technology leaders need to consider their entire technology ecosystem when assessing their supply chain risk. As threat actors have shown they can infiltrate hardware vendors, global software corporations, and open-source code repositories, organizations need a comprehensive security strategy. A supply chain attack could come from multiple vectors, and enterprises need to ensure they cover all their bases.

Software supply chain security 

Mitigating the risk of a supply chain attack requires a defense-in-depth approach. Organizations need to conduct thorough security assessments and implement multiple measures to minimize this risk. For example, many regulatory security frameworks such as PCI DSS and the NIST Cybersecurity Framework mention supply chain risk. These compliance standards state that organizations should routinely assess third parties to ensure they comply with any contractual obligations.

As part of the contractual obligations organizations enforce on their suppliers, security testing must form a vital part of any technology deliverable. By placing the responsibility on the vendor to ensure their product is safe, organizations can enforce terms should the vendor fail to meet their obligations. In addition to requiring vendors to test, organizations should also implement internal security testing and monitoring. This layered defensive approach can help them identify any security issues the vendors may have missed.

However, when leveraging open-source software, enforcing contractual terms is not an option. As many open-source technologies come with set licensing terms, it compounds the problem even further. Organizations also need to consider the license and infringement risk, restricting how the company can use the software while protecting itself from a supply chain attack. In these instances, leveraging the services of a Software Composition Analysis (SCA) tool like MergeBase mitigates this open-source risk. As the onus is on the organization to test and validate the open-source components used in an application, an SCA tool like MergeBase adds the required defensive layer.

According to this Gartner Report, the information security of a supply chain must focus on data and IT infrastructure, products, and operations. If we consider the various security technologies in place at enterprises today, organizations implement a myriad of defensive technologies and processes. Firewalls, intrusion detection and prevention solutions, segmented networking, and vulnerability scanners are just some of the solutions that protect their IT landscape. However, these solutions typically safeguard against external threats. Organizations need to ensure the configuration of these platforms also scan and detect any internal anomalies which may indicate a successful supply chain attack.

Enterprises can also consider air gapping systems to mitigate the supply chain risk. Applications and networks not connected to the Internet have a much lower risk of compromise. However, in many cases, this approach is not feasible. They are also not foolproof. The Stuxnet example mentioned earlier was a successful attack against an air-gapped system.

Securing the software supply chain

Supply chain attacks target less secure elements in complex systems. As modern technology solutions rely on reusable components, threat actors target these elements to circumvent security controls. This type of cyber-attack is not new. However, recent discoveries have highlighted the risk organizations face. Traditionally, organizations have tailored their security solutions to mitigate external threats. With the increase in supply chain attacks, it is clear that threat actors target the supply chain to circumvent these controls.

The supply chain attack risk covers every element of an organization’s IT landscape. Attackers have succeeded in infiltrating the supply chain of hardware and software elements. With software development components being a key area of risk, organizations need to implement controls that ensure the security of their application ecosystem.

The MergeBase platform specializes in software supply chain security. It mitigates the risk of a software supply chain attack from development to production. It highlights risks and empowers developers to remedy any security issues during the early stages of the software development lifecycle. MergeBase also assesses software components for vulnerabilities during the build process, ensuring organizations do not release insecure code to production. However, once an application is in production, it may not remain secure. Researchers or threat actors discover software vulnerabilities all the time. MergeBase mitigates this risk with its monitoring and alerting capabilities keeping organizations protected against any new vulnerabilities in production.

What is a supply chain attack?

Summary

harbour part of a traditional supply chain

A supply chain attack leverages the access of an external partner or provider to gain unauthorized entry to a system or network. It takes advantage of the inherent trust a target has in its suppliers, using it to infiltrate and launch the cyber-attack. Its use of stealth and its indirect ‘trusted’ approach make a supply chain attack an effective weapon in any threat actor’s arsenal.

Every organization and individual relies on third-party software in some way or another. When we install software, hardware, or use code from a trusted source, it is natural to assume that it hides no malicious intent. In addition to this inherent trust, a supply chain attack also uses the human element to bypass any perimeter security. As administrative users and software developers install software, hardware, or reuse third-party code, they do so on the internal network, bypassing any security controls that prevent external threats.

Supply chain attacks that target technology infrastructure come in many variations. Threat actors can infiltrate a software provider and embed malicious code that infects end users when they install or access the product. Another effective supply chain attack technique is infecting software code repositories that software developers leverage to create systems. Finally, threat actors can also infiltrate and infect the embedded software that operates the hardware on networking equipment, servers, and end-user devices. 

What is a software supply chain?

Modern software development processes rely on code reuse to build systems rapidly and cost-effectively. By leveraging existing code, developers can quickly assemble a system with its needed components instead of coding the entire solution from a blank canvas. Typically, programmers either reuse internally developed software code or leverage third-party libraries and frameworks. These components and their dependencies form part of the software supply chain. In other words, a software supply chain is a list of elements that goes into or affects the code from development to production. 

Almost every software application or service we use today leverages a software supply chain. For example, Netflix and Uber use Node.js, an open-source, server-side JavaScript platform that is well suited for scalable applications. Another example is WordPress, a content management system used to run nearly 40% of the world’s websites. The list goes on, but it suffices to say that organizations leverage software supply chains everywhere. Besides leveraging the frameworks and platforms mentioned, software developers also use code libraries to build their solutions. Services like GitHub and StackOverflow are valuable resources where developers can find libraries, code snippets, and advice to help them create solutions. 

However, a software supply chain does not only pertain to software development. It can also refer to instances where organizations install and run third-party applications in their technology environment. For example, every organization leverages third-party software for email. It would be both inefficient and expensive to develop an in-house solution for this utility. The same goes for system monitoring, file sharing, security, and other commodities in a technology environment. All these third-party applications and the external code it uses in its custom-developed applications form part of an organization’s software supply chain.

The anatomy of a supply chain attack

A supply chain attack infects the third-party technologies organizations use. It then leverages this unauthorized access to infiltrate and attack their primary targets. Typically, supply chain attacks start when threat actors exploit a vulnerability to gain access to a supplier’s systems. Once they have gained entry, they embed malicious code into the supplier’s software or hardware with a particular payload. The threat actor then waits until the target organization or user runs the supplier’s infected software or installs its infected hardware. As this infiltration technique circumvents any perimeter security, its indirect attack methodology is highly effective. It is also successful in gaining access to secure environments as these attacks typically target less secure elements in the supply chain.

Supply chain attacks are not a new type of threat, but recent cases have raised their prominence in the public domain. If we look at past instances, the Target data breach where malware infected their Point of Sale systems occurred in 2013. In that instance, the attackers compromised the organization’s third-party refrigerator vendor to infect Target’s POS environment with malware that stole credit card details. Another significant example is the famous Stuxnet malware that nation-states used to sabotage Iran’s nuclear centrifuges in 2010. In this example, the attackers used the digital certificates of Realtek Semiconductor to make their malware look legitimate to system administrators and evade anti-virus.  

More recently, in the SolarWinds supply chain attack, threat actors deployed malware during a routine update that emanated from SolarWinds’ servers. Every organization that ran the update was subsequently compromised, including technology companies and secure government agencies. As a result of this attack, the United States sanctioned Russia, believing that the Kremlin played a role in this mass infiltration. Other recent supply chain attacks include the narrowly averted PHP backdoor and the Code Dev incident.

These supply chain attack examples show that this technique has successfully infected many organizations around the world. What is of particular concern is that these supply chain attacks also succeeded in highly secure environments. The examples also show the extensive ramifications of a successful attack. One undetected infection can affect thousands of users and organizations.

The open-source risk

Many organizations use open-source software in some way, shape, or form. With open-source code present in 90% of modern applications, this vital element in the software development ecosystem is vulnerable to a supply chain attack. The recent PHP case mentioned earlier is a prime example. Modern software applications reuse open-source libraries, frameworks, and code snippets. Threat actors target these components as they are typically less secure. 

The 2020 Sonatype State of the Software Supply Chain Report stated next-generation attacks increased by 430% in the preceding 12 months. Unlike commercial software, open-source relies on the community to ensure its security. However, it is up to the organizations that use the software to conduct regular analysis, security audits, and penetration tests. 

Technology supply chain risk

The technology supply chain includes hardware and software. Although the focus of this article has been on software supply chain attacks, organizations cannot ignore the hardware risk. Numerous examples of mobile devices arriving with embedded malware and compromised networking equipment used to breach secure networks highlight this threat. 

 These instances illustrate that business and technology leaders need to consider their entire technology ecosystem when assessing their supply chain risk. As threat actors have shown they can infiltrate hardware vendors, global software corporations, and open-source code repositories, organizations need a comprehensive security strategy. A supply chain attack could come from multiple vectors, and enterprises need to ensure they cover all their bases. 

Mitigating supply chain risk

Mitigating the risk of a supply chain attack requires a defense-in-depth approach. Organizations need to conduct thorough security assessments and implement multiple measures to minimize this risk. For example, many regulatory security frameworks such as PCI DSS and the NIST Cybersecurity Framework mention supply chain risk. These compliance standards state that organizations should routinely assess third parties to ensure they comply with any contractual obligations. 

 As part of the contractual obligations organizations enforce on their suppliers, security testing must form a vital part of any technology deliverable. By placing the responsibility on the vendor to ensure their product is safe, organizations can enforce terms should the vendor fail to meet their obligations. In addition to requiring vendors to test, organizations should also implement internal security testing and monitoring. This layered defensive approach can help them identify any security issues the vendors may have missed. 

However, when leveraging open-source software, enforcing contractual terms is not an option. As many open-source technologies come with set licensing terms, it compounds the problem even further. Organizations also need to consider the license and infringement risk, restricting how the company can use the software while protecting itself from a supply chain attack. In these instances, leveraging the services of a Software Composition Analysis (SCA) tool like MergeBase mitigates this open-source risk. As the onus is on the organization to test and validate the open-source components used in an application, an SCA tool like MergeBase adds the required defensive layer.

According to this Gartner Report, the information security of a supply chain must focus on data and IT infrastructure, products, and operations. If we consider the various security technologies in place at enterprises today, organizations implement a myriad of defensive technologies and processes. Firewalls, intrusion detection and prevention solutions, segmented networking, and vulnerability scanners are just some of the solutions that protect their IT landscape. However, these solutions typically safeguard against external threats. Organizations need to ensure the configuration of these platforms also scan and detect any internal anomalies which may indicate a successful supply chain attack.

Enterprises can also consider air gapping systems to mitigate the supply chain risk. Applications and networks not connected to the Internet have a much lower risk of compromise. However, in many cases, this approach is not feasible. They are also not foolproof. The Stuxnet example mentioned earlier was a successful attack against an air-gapped system. 

Securing the software development supply chain

Supply chain attacks target less secure elements in complex systems. As modern technology solutions rely on reusable components, threat actors target these elements to circumvent security controls. This type of cyber-attack is not new. However, recent discoveries have highlighted the risk organizations face. Traditionally, organizations have tailored their security solutions to mitigate external threats. With the increase in supply chain attacks, it is clear that threat actors target the supply chain to circumvent these controls. 

The supply chain attack risk covers every element of an organization’s IT landscape. Attackers have succeeded in infiltrating the supply chain of hardware and software elements. With software development components being a key area of risk, organizations need to implement controls that ensure the security of their application ecosystem.

The MergeBase platform mitigates the risk of a software supply chain attack from development to production. It highlights risks and empowers developers to remedy any security issues during the early stages of the software development lifecycle. MergeBase also assesses software components for vulnerabilities during the build process, ensuring organizations do not release insecure code to production. However, once an application is in production, it may not remain secure. Researchers or threat actors discover software vulnerabilities all the time. MergeBase mitigates this risk with its monitoring and alerting capabilities keeping organizations protected against any new vulnerabilities in production.

Open Source Risk: Plugging the hole

"The leak was worse than first thought" notes a bypasser of a boy stuck head-first into the wall of a digital dyke. Plug you open source holes with a SCA tool

Origin 

Software development based on the sharing and collaborative improvement of software source code goes back to its very origins. 

In the late 1990s, the term “open-source” was coined and received mainstream recognition in publications such as Forbes. The Netscape browser’s source code was made open source and that got a lot of attention.

The original open-source projects were “revolutions” against the “unfair” profits that closed-source software companies were reaping. Microsoft, Oracle, SAP and others, it was argued, were extracting monopoly-like “rents” for software, which the top developers of the time did not believe was world class.

Open Source Growth

Open source software was originally created by developers for developers. It was embraced slowly by more and more projects, organisations and companies and it now forms the foundation for the Internet and most of our digital assets. The code base of a typical modern application consists of 80 to 90% of open source software. Even in something as proprietary as Apple’s iPhone, the operating system consists largely of open source software. 

Currently, there are close to 1 million open source projects globally and this number increases by 79% a year

Open source victorious as last ones standing capitulate

Apple and Google embraced open source more than 20 years ago. The champions of proprietary software, IBM and Microsoft, resisted much longer. 

  “Once open source gets good enough,
competing with it would be insane.”

2006, Larry Elison, the chairman of Oracle in conversation with the Financial Times

Elison was right on the mark. It looks like we reached that point a few years ago. IBM and Microsoft were the last ones standing against open source, but  in the end they capitulated. IBM acquired RedHat  early 2019 for $34B and Microsoft acquired GitHub for $7.5B in 2018.

Open source use a surprise to many executives

Many organizations where leadership does not have a strong engineering or technical background often do not fully realize yet the importance of open source and how dependent they are on it in their digital supply chain. We regularly encounter executives who are very surprised when we analyze their applications and identify many open source libraries. Awareness is the first step in managing open source risk and rewards.

Open Source Risks: Is it really free?

Open source is bringing huge rewards to business. However, with reward comes open source risk. The two main risks are legal related to the licenses  and cyber risk related to vulnerabilities. 

Open source is free but can come with strings attached that do not match with your organization’s business model. Open source software is released under different licensing models. There are over 300 licensing models in use. Most open source software comes with friendly licenses such as the licenses for Apache and BSD. However other licensing models not so much, such as licenses for GNU GPL and GNU Affero. Use of these licenses, even in a minor way, could force an organisation to open source their entire software with devastating impact on the IP value of the organisation.  

Open source software, like all software, can contain vulnerabilities. Open source software, in general, is high quality software and not intrinsically more vulnerable. However, because of its wide usage, it is a very attractive target for cyber adversaries and so, over time, vulnerabilities are uncovered. At the moment, there are more than 150,000 known vulnerabilities. A lot of these vulnerabilities can be exploited to breach organisations and are considered to be the cause of approximately 25% of data breaches. 

One example of a major breach is the Equifax breach which exposed 145 million client records and cost the organisation more than $1.3 B to remediate. The company also lost $5B in stock market value overnight and later received a $700 M fine from the US government. 

The best defence: SCA / OSS

The best defense against open source risk is to use a Software Composition Analysis tool, sometimes also called Open Source Security scanner. These tools quickly analyse your applications or containers and provide insight into license and cyber risk. MergeBase goes a step further and provides solutions to quickly and easily reduce your cyber risk. 

A Critical Look at Cyber Investment

What is the top defensive technology area to invest in right now?

Cyber defense is a global whack-a-mole game with hundreds of billions of dollars being invested in offensive and defensive capabilities. After you invest in one area, another area of risk tends to pop up. What is the top defensive technology area to invest in right now?

Cyber is multifaceted

Cyber defense requires a multifaceted approach. Fragmentation is a natural consequence of the back and forth between cyber attackers and defenders: If we have an effective defence against a particular type of attack, adversaries will try another area, angle, or approach. Over time this means we need many technologies to secure our organisation. Like it or not, cyber defence is a global whack-a-mole game. It is an arms race, with governments and corporations investing hundreds of billions of dollars continuously in building out offensive and defensive capabilities.

We all know that we need a multifaceted approach. This involves people, process and tools. We need to make sure that everyone in the organization is motivated and has the skills and resources to fight cybercrime. Beyond understanding why and how, technology is critically important as cyberspace is tech heavy.

What area do we need to invest in?

Unless you feel at ease with your cyber protection, the question is: What is the key technology area to invest in right now? This question is very difficult for most cyber professionals as most organizations under fund and under resource their cyber operations.

We posed this question to cyber professionals by posting a poll to LinkedIn. To eliminate bias, we conducted the poll twice (second poll), reaching out to two distinct networks of cyber professionals. Feel free to repost the poll and let us know what your results are.

The poll asked what areas to focus on: MFA, perimeter security, known vulnerabilities or education. The results, which were consistent between the two polls, were: known vulnerabilities at 49% , MFA at 29%, and perimeter and education each approximately at 10%.

Known vulnerabilities routinely exploited

The results of the poll make a lot of sense. Of course, all these areas are important and really need more investment. However, the NSA and CISA continue to warn that cyber adversaries routinely exploit known vulnerabilities..

If we look at major breaches, we see plenty of evidence supporting these warnings. Sophisticated attackers use a combination of hacking techniques, as we have seen recently with SolarWinds. Exploiting known application vulnerabilities is a big part of their arsenal and allows adversaries to move laterally and subsequently elevate privileges.

In reality we find that very few organizations are able to execute fully on a vulnerability strategy.

Why can we not eliminate known vulnerabilities?

Why are we not able to routinely eliminate our known application vulnerabilities? The answer is that it is a daunting task given the level of software that most organisations are operating in combination with the level of technical debt that most of these applications suffer from. Some cyber experts call for continuous upgrading of all components. That would eliminate these problems. However, continuous upgrading is difficult for organisations that have a lot of applications. For instance, a typical North American bank has 600 software applications. Large banks tend to have many more. A lot of these applications are older and do not have active development. Therefore, routinely upgrading may not be practical.

Software Composition Analysis (SCA) vs. Java Über Jars

Image source: unsplash.com.

Introduction

Über jars are a type of reuseable Java library that applications sometimes (knowingly or not) incorporate into their systems. Über jars are particularly challenging for software composition analysis (SCA) tools to understand because their structure and organization is complex. In this blog post I explain what über jars are, why they exist, and I provide a mini-benchmark to see how current SCA tools deal with this type of Java library.

TLDR

Circa February 2021, both MergeBase and Sonatype use deep binary analysis to measure software composition within über jars.

  • MergeBase is the only SCA tool I observed with comprehensive support for über jars.
  • Sonatype also provides decent support but misses a few obvious cases. (Strangely, their legal analysis completely ignores their über jar results.)
  • OWASP Depedency-Check and JFrog Xray are not bad, but their scanning is based solely on metadata files found inside über jars.
  • Snyk, WhiteSource, and Github Dependabot currently have no ability to understand über jars at all.

I did not benchmark Black Duck (I don’t have access to that tool).

Background

Über jars are the Java equivalent of taking everything in your fridge and throwing it all into your largest pot, giving it a good stir. From an SCA (Software Composition Analysis) perspective they are a bit of a nightmare.

Recall: normally you point your SCA tool at a Java jar file (a reuseable Java library) and your SCA tool responds by telling you the jar file’s name, version, and known-vulnerabilities.

But what if that single Java jar file is actually an agglomeration of dozens of Java jar files? What if some maniac cracked open all your jar files and poured all their contents into a single mega jar? That’s exactly what an über jar is and your SCA tool is going to need to reverse-engineer the contents accurately before it can say anything.

Why would anyone create an über jar in the first place?

Java programs are awkward to invoke. You have to tell Java the exact locations of all the jar files your program is using. Über jars are a way around that problem.

For example, a typical normal java program is started like this:

java -classpath lib1.jar:lib2.jar:main.jar name.of.MainEntry

With an über jar it’s less typing, since “lib1.jar” and “lib2.jar” have been blended directly into a single “main-uber.jar” file:

java -classpath main-uber.jar name.of.MainEntry

In this way über jars make Java programs easier to distribute and easier to start. That’s the main reason why they exist.

Recall that Jar files are actually just zip files. You can rename them to “.zip” and then double-click on them if you ever want to see what’s inside them. Über jars are what you get if you unzipped all of your Jar files and combined all the contents into a single zip file instead.

Über jars are a challenge for SCA

Most SCA tools are geared towards providing a single succinct answer for each library they scan.

SCA Scan Results

Identified Library:
Apache Commons-Collections 3.2.1.

Vulnerabilities:
CVE-2017-15708, CVE-2015-7501, and CVE-2015-6420

With über jars the answer is more complicated. “Well, actually… this library is a combination of many libraries.”

At MergeBase we analyze every jar file against our master database for this possibility. For example, consider “apacheds-all-1.5.5.jar”, a large über jar containing over 500,000 lines of code coming from dozens of libraries. When we compare this jar file against all known versions of “slf4j-api” here are the results:

Match RatioKnown Library Version
81.0%slf4j-api@1.5.11
90.5%slf4j-api@1.5.8
100.0%slf4j-api@1.5.6
90.5%slf4j-api@1.5.5
90.5%slf4j-api@1.5.4
These results show that version 1.5.6 of slf4j-api is contained inside the apacheds-all-1.5.5 über jar file.

In the “slf4j-api” case there is also another hint inside the über jar. If I grep the jar’s contents for “sl4fj-api” I see these two entries:

    META-INF/maven/org.slf4j/slf4j-api/pom.xml
    META-INF/maven/org.slf4j/slf4j-api/pom.properties

Opening the latter, I see this:

    #Generated by Maven
    #Fri Nov 21 14:48:07 CET 2008
    version=1.5.6
    groupId=org.slf4j
    artifactId=slf4j-api

This gives me further confidence that my binary analysis is correct: version 1.5.6 aligns with my MergeBase result. Some SCA scanners only consider this metadata when examining über jars, but philosophically I don’t agree with that approach, since metadata is not always present, as in the bouncy-castle example below. Metadata is also vulnerable to transcription mistakes and tampering.

You might be curious why this metadata is even present in the first place.

My own theory: it was probably present in the original “slf4j-api” jar. Über jars don’t just combine the software files – they combine all the files! And so if a metadata file is present in the original “slf4j-api” file, it will be dutifully copied into the über jar. I can download the original and see for myself. Sure enough, running “unzip -l slf4j-api-1.5.6.jar” shows both of those metadata files were in the original.

Moving onto to an example without metadata, here’s the results when we compare our über jar against “bcprov-jdk15”:

Match RatioKnown Library Version
84.7%bcprov-jdk15@1.44
91.3%bcprov-jdk15@1.43
100.0%bcprov-jdk15@1.40
82.0%bcprov-jdk15@1.38
48.6%bcprov-jdk15@1.32

There is no metadata available to warn consumers that the highly vulnerable version 1.40 of bcprov-jdk15 was copied into apacheds-all-1.5.5.jar. Unfortunately bcprov-jdk15@1.40 contains over 15 known-vulnerabilities. Scanners that rely on metadata (such as JFrog Xray and OWASP Dependency-Check) will miss this. And of course scanners that lack über jar handling (such as WhiteSource and Snyk) will also miss this.

Using our high-confidence matches we then query our known-vulnerability database for any corresponding vulnerabilities. Our technique is based on binary analysis – no metadata is involved at all, since metadata can be inaccurate. Using this technique we are able to identify dozens of sub-components encapsulated by the apacheds-all-1.5.5 über jar. Here’s a partial listing based on MergeBase’s analysis:

  1. 100.0% – antlr/antlr@2.7.7
  2. 100.0% – commons-io/commons-io@1.4
  3. 100.0% – commons-lang/commons-lang@2.4
  4. 100.0% – org.apache.directory.server/apacheds-core-jndi@1.5.5
  5. 100.0% – org.apache.directory.shared/shared-ldap@0.9.15
  6. 100.0% – org.apache.mina/mina-core@2.0.0-M6
  7. 100.0% – org.bouncycastle/bcprov-jdk15@1.43
  8. 100.0% – org.slf4j/slf4j-api@1.5.8

(Etc… 25 more sub-components identified!)

Quick Competitive Check

We were curious to see if competing SCA tools are able to handle über jars. What follows is a quick benchmark against a half-dozen popular SCA tools.

Methodology

For each SCA tool (MergeBase, OWASP Dependency-Check, Snyk, WhiteSource, Sonatype, etc…):

  1. Git clone: https://github.com/mergebase/vuln-example-apacheds-all
  2. Run “mvn install”.
  3. Apply each SCA tool against the built “vuln-example-apacheds-all”.
  4. Observe and compare the scan results.

Mini-Benchmark Results

As of February 2021, the apacheds-all-1.5.5 über jar contains two vulnerable sub-components. One of these (bcprov-jdk15@1.40) can only be identified using binary approaches since it had no metadata in the first place, and one of these (commons-collections@3.2.1) can be identified either via binary approaches or via metadata scanning.

We group the benchmark results into 3 categories:

1. Scanners that do not support über jars at all.

Snyk and Whitesource appear to have no idea that “apacheds-all@1.5.5” is made by combining many jar files together.

Similarly, Github’s Dependabot also has no idea about this.

2. Scanners that support a metadata-based understanding of über jars.

OWASP Dependency-Check and JFrog Xray both detect the “commons-collections@3.2.1” metadata inside the über jar.

3. Scanners that support deep understanding of über jars.

Sonatype fails to identify any known-vulnerabilities with respect to commons-collections@3.2.1, and yet it does correctly identify that apacheds-all@1.5.5 contains bcprov-jdk15@1.40! This is a lopsided result: Sonatype clearly has a deep understanding here (otherwise it would be impossible to identify bcprov-jdk15), and yet somehow Sonatype is failing to spot the easy one. We also noted that Sonatype reported the license as Apache 2.0, when bcprov-jdk15 uses the MIT license.

MergeBase identifies all vulnerabilities correctly in this case. 🙂

Conclusion

Über jars are a special type of Java software component made by combining several jars into a single jar. Aside from MergeBase, most SCA scanners currently provide sub-par or even zero support for this component type.

Special Thanks

Specials thank to Dr. Ken Warkentyne, our principal engineer, who built MergeBase’s über jar scanning capabiliity.

Scanning .NET and Nuget projects for known vulnerabilities

We recently (August 2020) completed version 1 of our .NET scanner. Its goal is scanning .NET and Nuget projects for libraries with known vulnerabilities in any .NET project.

For this blog post we thought we’d take our scanner out for a spin and see how it compares against the competition.

TL;DR:

Here are the results of scanning .NET and Nuget projects for known vulnerabilities:

  • MergeBase – 18 vulnerabilities, 0 false positives.
  • Snyk – 7 vulns and 5 false, or 4 vulns and 0 false (depends on scanner setup).
  • WhiteSource – 12 vulns, 0 false.
  • OWASP Dependency Check – 12 vulns, 17 false.
  • Dotnet Retire – 2 vulns, 0 false.
  • Sonatype – 0 vulns, 0 false.
  • Dependabot – 0 vulns, 0 false.

Methodology:

We chose the .NET Orleans project as to scan for .NET and Nuget vulnerabilities. It’s active, complex, and it builds successfully (August 6th, 2020, master = 2e10856f7b7ed9443c). We also liked how this project contained a mix of Nuget styles (e.g., older “packages.config” style as well as the newer “<PackageReference/>” style).

We type “dotnet build” before scanning. This way .NET scanners can use the generated “obj/project.assets.json” files to supplement their scan data if they want to, and “dotnet build” is such a critical step for building any .NET project that we think it’s safe for an SCA tool to assume this command has completed successfully.

As for comparing results, we count CVE’s. If the scan outputs 1 or 300 or 9,000,000 hits against CVE-2018-8292, we count that as a single CVE. We then do a quick “desk check” to categorize the result as either a true-hit, a false-negative, or an ambiguous result (where it’s hard to say one way or the other). The “desk check” is very much based on my own decades of experience as a software engineer – I encourage others to rerun these scans and see if they agree or disagree.

Because this is a .NET scan for Nuget project, we ignore any results the scanners find from other file-types lying on the file system (e.g., “VotingWeb/wwwroot/lib/jquery/jquery.min.js”). We do, however, count results found from nuget references into other language artifacts (e.g., “GPSTracker.Web/packages.config” contains a nuget reference to “<package id=”bootstrap” version=”3.0.0″ targetFramework=”net45″ />” in its packages.config file – we’ll count this.)

Here is the exact sequence of steps:

  1. git clone https://github.com/dotnet/orleans.git
  2. dotnet build
  3. Deploy The Scanners!
  4. Validate the results.

A note about ambiguous results:

We classify some results as ambiguous. This means there’s definitely some smoke, so we can’t rule immediately it out as a false negative after examining the metadata, but on the other hand, there’s enough uncertainty to also make us uncomfortable considering it a true hit.

Example:

The vulnerability references “bootstrap” in the scan report but the CVE description talks about “bootstrap-sass”. Maybe? Or in another case the CVE description starts out with the words (in all caps) “DISPUTED”.

Results of scanning .NET, Nuget project for vulnerabilities:

I’ll save the best for first! Here’s what MergeBase finds:

1. MergeBase

18 vulnerabilities found (and two ambiguous hits).

Drop the scanner into the Orleans subdirectory. Type “java -jar mergebase.jar .” and the results are pretty straightforward: 2 critical CVE’s, 5 high ones, and 11 mediums. A quick spot-check of the metadata looked good (no false positives and two ambiguous results).

2. Github Dependabot

Zero vulnerabilities found.

Dependabot not doing too much here, despite being a Microsoft product (albeit recently acquired):

3. Dotnet Retire

Two vulnerabilities found: CVE-2018-8292 and CVE-2018-8416. MergeBase also found these two among the 18 vulnerabilities it identified.

4. Sonatype

Zero .NET vulnerabilities found!

Sonatype does detect a small handful of JavaScript vulnerabilities (since Orleans contains things like “VotingWeb/wwwroot/lib/jquery/jquery.min.js”), but nothing for .NET. To be fair, their scanner instructions did say “you must copy all .NET packages you depend on into the zip file you are scanning beforehand.” I typed “dotnet build” and zipped the result (660MB). As far as I’m concerned, I was doing them a favour by even zipping up orleans post-build in the first place – no other scanner required that.

Note-to-self: Probably MergeBase should also scan those JavaScript packages! (Our current logic looks for NPM and Yarn lock files, but maybe it’s time to roll up our sleeves and consider scanning raw *.js and *.min.js files, too.)

5. Snyk

“It’s Complicated!” The problem with Snyk is that there’s two different ways to invoke the Snyk scanner, and each way returns wildly different results.

Snyk Approach #1 – Github Integration:

15 vulnerabilities found. 5 of those are false positives (all because Microsoft.NETCore.App was flagged as a dependency, but it’s not). 3 are ambiguous. 7 are true hits.

A few NPM and Docker vulnerabilities also found, but seeing as this bakeoff is only about .NET we ignored those.

Snyk Approach #2 – Command Line Invocation:

7 vulnerabilities found. 3 are ambiguous, leaving 4 true hits, including 1 true hit that Snyk approach #1 above did not find (CVE-2020-1469).

No NPM or Docker vulnerabilities found via this approach.

6. Whitesource Bolt

12 true CVE vulns.

3 ambigs.

7. OWASP Dependency Check

12 true CVE vulns.

3 true NON-CVE vulns.

2 ambigs.

17 falses

Unfortunately OWASP Dependency Check is currently unable to handle .NET’s property substitution (e.g., when a *.csproj file references “Directory.Build.props”), a common convention for developers maintaining these files. This causes some frustrating false positives, such as reporting that “Google.Protobuf:$(GoogleProtobufVersion)” is vulnerable to CVE-2015-5237.

OWASP Dependency Check also considers version 0.61.0 of the .NET MySqlConnector package to be vulnerable to 14 CVE’s – these are certainly all false positives. This is probably happening because Dependency Check considers version “0.61.0” to come before releases from MySQL’s popular version 5.x series against which many CVE’s have been filed over the years. However, version “0.61.0” of this package is less than 10 months old, making it impossible that it’s vulnerable to these ancient CVE’s.

Scanning .NET and Nuget Conclusion

Our own offering looks compelling in the .NET space. We were one of the top performers in this mini-benchmark, with Snyk, Whitesource, and the open source OWASP Dependency Check tool also providing reasonable results. We were surprised to see Sonatype and Dependabot perform so poorly here. Software developers currently using the popular open source “Dotnet Retire” tool for this problem should definitely consider other options.

As always with benchmarks such as these and security tools in general your mileage can vary a lot based on the tools you’re using and your own particular context. I think a lot of companies become complacent with their existing tools. Similar to with the smoke detectors in your house, it’s a good idea to benchmark your existing tools periodically, just to ensure that they are still working properly!

Last piece of advice: Have .NET software? Give MergeBase a closer look!

Introducing CodeGreen for Bitbucket

Recommended pre-reading:
  
Intro to SCA – Software Composition Analysis (mergebase.com)

Atlassian Marketplace Link:
MergeBase CodeGreen (marketplace.atlassian.com)

Introduction

One of the main challenges with known-vulnerabilities is how they mess with standard software lifecycles. A lot of traditional quality engineering relies on the old saying, “if it’s not broken, don’t touch it.” Known-vulnerability announcements for popular open source libraries completely go against that, since they are discovered and announced more or less at random. A good known-vulnerability SCA solution needs to deal with three very different cadences through which known-vulnerabilities will manifest themselves in your software:

If you’re serious about reducing open-source known-vulnerabilities within your software assets, CodeGreen is a tool for getting real results company wide. CodeGreen puts known-vulnerability software composition analysis (SCA) scans directly in front of software engineer eyeballs. A lot of application security work is done by following checklists and invoking security tools and uploading artifacts to cloud URLs during coding and reviewing tasks. CodeGreen short circuits all that by inserting itself directly into your company’s software engineering workflow (as a Bitbucket plugin). From there CodeGreen can inject a range of interventions customized to your corporate application security policy, from low-friction informational reports all the way to outright blocking. These interventions help you quickly get all of your software engineering teams onto the same page.

By attaching directly to the enterprise source-control system (as a Bitbucket plugin) GodeGreen is able to improve application security posture across the board for an entire organization. Your application security will improve within hours after your local Bitbucket administrator installs the CodeGreen plugin through Atlassian’s marketplace.

Vulnerabilities Arrive On Different Cadences

  1. New vulnerability announcements. Your application is not broken, in fact it’s working great! Clients love it. Management is happy. But a known vulnerability has been discovered and published that could be exploited by criminals and bring your brand down. You have to fix it! You must upgrade the insecure library to a safer version.
  2. Accidental vulnerability import (“developer-as-vector”). Under this scenario one of your developers unwittingly introduces a bad library version (that contains known-vulnerabilities) into one of your systems. Just because a “known-vulnerability” is known to the cyber security world at large does not mean it’s known to your own development staff!
  3. That terrifying first scan. This scenario is essentially a combination of the above two scenarios, albeit after several years of unmonitored vulnerability accumulation. The experience of running a first vulnerability scan can be so overwhelming and demoralising for staff that good SCA tools must account for this and provide strategies to manage the first scan.

CodeGreen is a unique tool in the SCA space in that it provides mitigations, reports, and controls designed specifically for these 3 cadences. The rest of this blog post goes into those capabilities in-depth.

For New-Vulnerability-Announcements: Add A Little Friction (Cadence #1)

Developers need to be aware of how newly discovered vulnerabilities affect their systems, but finding time to address these is always a balancing act based on risk, urgency, and other priorities. This is where CodeGreen can apply a little friction.

For Developer-As-Vector: Slam On The Brakes! (Cadence #2)

For cadence #2 (developer-as-vector), once awareness is in place, vulnerabilities should never come into software via this vector. The vast majority of software vulnerabilities are announced alongside a patched (fixed) release of the library. This means developers should never introduce vulnerable libraries into a software project unless such is absolutely unavoidable. This is where CodeGreen can slam on the brakes.

Managing That Terrifying First Scan (Cadence #3)

A lot of security tools are sold and marketed based on a simplified models of their operation – the tool is presented similar to a flashlight. Turn on the light, see into the darkness. But under the hood the tool might offer dials and controls and subtleties to users to help make its operation more successful. CodeGreen is no exception here!

Under ideal operation CodeGreen would be configured to apply maximum friction to encourage developers to eliminate all vulnerabilities, but that’s not tractable for most organizations, at least not at first.

To help make CodeGreen more practical we allow repository administrators to adjust the CVSS thresholds at which the various CodeGreen mitigations become active:

We recommend setting these to more permissive values during your initial rollout, and tightening them to more restrictive values as your teams’ application-security maturity improves.

For example, in the beginning you might want to enable only the CodeGreen double-push friction and set it to a CVSS 9.0 threshold and disable everything else. Make it an overt term goal to clear out all 9.0 vulnerabilities and above.

(But always enable “block-net-new-vulnerabilities” because that’s the dreaded cadence #2!)

Once you’ve achieved that, increase the “double-push” control to use a CVSS threshold of 8.0, so it catches more vulnerabilities.

Meanwhile, enable the “requires dual-approval” control (a much higher friction compared to double-push) and set that one to 9.0.

The end result here is interesting: any newly announced vulnerabilities will suddenly dramatically slow down development teams. The developer has a choice: find someone to approve their work, leaving the vulnerability in place, or just patch the brand new 9.8 vulnerability and avoid the dual-approval.

Which would you choose?

It’s a lot like thoroughly cleaning a house methodically from top to bottom: once a given room is clean, you can lock its door to prevent any additional mess from occurring in the already cleaned room. Similarly here you can clean out all the 9.0’s and above, and then “lock the door” on them by turning on the dual-approval control.

Conclusion

GodeGreen improves application security posture across the board for your entire organization by embedding open-source known-vulnerability scans directly into your centralized git source control. Your application security will improve within hours after your local Bitbucket administrator installs it!

Discover More from MergeBase

Core Product

BuildGreen is a powerful solution for identifying the real risk of open source at build time or in existing applications

Learn how BuildGreen can protects your Enterprise

Add RunTime Protection

RunGreen detects and defends against known-vulnerabilities at runtime.

Learn why Runtime Protection Matters

Optional Developer Add-on

CodeGreen is an early-warning defence for your in-house development and integrates directly into code repositories

Quick Start - For Free