Scanning a firecracker microVM with MergeBase

Firecracker microVM is a virtual machine monitor that allows you to create and manage microVMs. It leverages the Linux Kernel-based Virtual Machine (KVM) and utilizes a minimalist design for increased security. As firecracker microVMs do not include unnecessary devices and guest functionality, they provide a reduced memory footprint and attack surface area. 

The firecracker architecture is used by and integrated with several infrastructure solutions such as appfleet, containerd, fly.io, and OpenNebula. In this article, we will be building a firecracker containerd microVM and scan it for any known vulnerabilities with MergeBase.

In this article:

Building a firecracker microVM

As firecracker leverages KVM, you need to run it on a bare metal server that supports virtualization, such as an AWS i3.metal instance. You can run it on a virtual machine that utilizes nested virtualization. However, this platform is not supported. The following script tests your current environment and will let you know if it is capable of running a firecracker.  

#!/bin/bash
err=""; \
[ “$(uname) $(uname -m)” = “Linux x86_64” ] \
 || err=“ERROR: your system is not Linux x86_64.”; \
[ -r /dev/kvm ] && [ -w /dev/kvm ] \
 || err="$err\nERROR: /dev/kvm is innaccessible."; \
(( $(uname -r | cut -d. -f1)*1000 + $(uname -r | cut -d. -f2) >= 4014 )) \
 || err="$err\nERROR: your kernel version ($(uname -r)) is too old."; \

dmesg | grep -i “hypervisor detected” \
 && echo “WARNING: you are running in a virtual machine. Firecracker is not well tested under nested virtualization.”; \

[ -z “$err” ] && echo “Your system looks ready for Firecracker!” || echo -e “$err”

If your system can run firecracker microVM, the script will confirm it, as illustrated in the output below.

root@firecracker:~# bash test.sh
[ 0.000000] Hypervisor detected: KVM
root@firecracker:~#

Install the latest version of Go

The firecracker microVM team has created a quick start guide you can use to get your firecracker-containerd instance up and running. However, if you run the script, it will fail as it uses Go modules that leverage later versions of the Go programming language. The workaround for this challenge is to install the latest version of Go manually, as illustrated in the steps below.

Download the latest version of Go

cd /tmp
wget https://golang.org/dl/go1.<\Version_Number>.linux-amd64.tar.gz

Extract the tarball to the installation location – /usr/local

tar -C /usr/local -xzf go1.<Version_Number>.linux-amd64.tar.gz

Set the environment

_echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.profile_ 
echo "export GOPATH=~/.go" >> ~/.profile 
source ~/.profile

 If you have installed the latest version of Go successfully, running the command go version will verify it, as shown in the output below.

root@firecracker:/tmp# go version
go version go1.16.5 linux/amd64
root@firecracker:/tmp#

Download and install the required dependencies on a Debian-based instance

Now that you have the latest version of Go installed, you can run the script below provided by the firecracker team that downloads and installs the required dependencies on a Debian based instance.

#!/bin/bash
 
cd ~
 
# Install git, Go 1.13, make, curl
sudo mkdir -p /etc/apt/sources.list.d
echo "deb http://ftp.debian.org/debian buster-backports main" | \
  sudo tee /etc/apt/sources.list.d/buster-backports.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
  install --yes \
  golang-1.13 \
  make \
  git \
  curl \
  e2fsprogs \
  util-linux \
  bc \
  gnupg
 
# Debian's Go 1.13 package installs "go" command under /usr/lib/go-1.13/bin
export PATH=/usr/lib/go-1.13/bin:$PATH
 
cd ~
 
# Install Docker CE
# Docker CE includes containerd, but we need a separate containerd binary, built
# in a later step
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt-key finger docker@docker.com | grep '9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88' || echo '**Cannot find Docker key**'
echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
     install --yes \
     docker-ce aufs-tools-
sudo usermod -aG docker $(whoami)
 
# Install device-mapper
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y dmsetup

Download and install firecracker-containerd

Once you have downloaded and installed the dependencies, you can move on to the next step and get firecracker-containerd downloaded and installed. Again, we will use the script below provided by the firecracker team.

#!/bin/bash
cd ~
 
# Check out firecracker-containerd and build it. This includes:
# * firecracker-containerd runtime, a containerd v2 runtime
# * firecracker-containerd agent, an inside-VM component
# * runc, to run containers inside the VM
# * a Debian-based root filesystem configured as read-only with a read-write
# overlay
# * firecracker-containerd, an alternative containerd binary that includes the
# firecracker VM lifecycle plugin and API
# * tc-redirect-tap and other CNI dependencies that enable VMs to start with
# access to networks available on the host
git clone https://github.com/firecracker-microvm/firecracker-containerd.git
cd firecracker-containerd
sg docker -c 'make all image firecracker'
sudo make install install-firecracker demo-network
 
cd ~
 
# Download kernel
curl -fsSL -o hello-vmlinux.bin https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux.bin
 
# Configure our firecracker-containerd binary to use our new snapshotter and
# separate storage from the default containerd binary
sudo mkdir -p /etc/firecracker-containerd
sudo mkdir -p /var/lib/firecracker-containerd/containerd
# Create the shim base directory for which firecracker-containerd will run the
# shim from
sudo mkdir -p /var/lib/firecracker-containerd
sudo tee /etc/firecracker-containerd/config.toml <<EOF
disabled_plugins = ["cri"]
root = "/var/lib/firecracker-containerd/containerd"
state = "/run/firecracker-containerd"
[grpc]
  address = "/run/firecracker-containerd/containerd.sock"
[plugins]
  [plugins.devmapper]
    pool_name = "fc-dev-thinpool"
    base_image_size = "10GB"
    root_path = "/var/lib/firecracker-containerd/snapshotter/devmapper"
 
[debug]
  level = "debug"
EOF
 
# Setup device mapper thin pool
sudo mkdir -p /var/lib/firecracker-containerd/snapshotter/devmapper
cd /var/lib/firecracker-containerd/snapshotter/devmapper
DIR=/var/lib/firecracker-containerd/snapshotter/devmapper
POOL=fc-dev-thinpool
 
if [[ ! -f "${DIR}/data" ]]; then
    sudo touch "${DIR}/data"
    sudo truncate -s 100G "${DIR}/data"
fi
 
if [[ ! -f "${DIR}/metadata" ]]; then
    sudo touch "${DIR}/metadata"
    sudo truncate -s 2G "${DIR}/metadata"
fi
 
DATADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/data)"
if [[ -z "${DATADEV}" ]]; then
    DATADEV="$(sudo losetup --find --show ${DIR}/data)"
fi
 
METADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/metadata)"
if [[ -z "${METADEV}" ]]; then
    METADEV="$(sudo losetup --find --show ${DIR}/metadata)"
fi
 
SECTORSIZE=512
DATASIZE="$(sudo blockdev --getsize64 -q ${DATADEV})"
LENGTH_SECTORS=$(bc <<< "${DATASIZE}/${SECTORSIZE}")
DATA_BLOCK_SIZE=128
LOW_WATER_MARK=32768
THINP_TABLE="0 ${LENGTH_SECTORS} thin-pool ${METADEV} ${DATADEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK} 1 skip_block_zeroing"
echo "${THINP_TABLE}"
 
if ! $(sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
    sudo dmsetup create "${POOL}" --table "${THINP_TABLE}"
fi
 
cd ~
 
# Configure the aws.firecracker runtime
# The long kernel command-line configures systemd inside the Debian-based image
# and uses a special init process to create a read-write overlay on top of the
# read-only image.
sudo mkdir -p /var/lib/firecracker-containerd/runtime
sudo cp ~/firecracker-containerd/tools/image-builder/rootfs.img /var/lib/firecracker-containerd/runtime/default-rootfs.img
sudo cp ~/hello-vmlinux.bin /var/lib/firecracker-containerd/runtime/default-vmlinux.bin
sudo mkdir -p /etc/containerd
sudo tee /etc/containerd/firecracker-runtime.json <<EOF
{
  "firecracker_binary_path": "/usr/local/bin/firecracker",
  "cpu_template": "T2",
  "log_fifo": "fc-logs.fifo",
  "log_levels": ["debug"],
  "metrics_fifo": "fc-metrics.fifo",
  "kernel_args": "console=ttyS0 noapic reboot=k panic=1 pci=off nomodules ro systemd.journald.forward_to_console systemd.unit=firecracker.target init=/sbin/overlay-init",
  "default_network_interfaces": [{
    "CNIConfig": {
      "NetworkName": "fcnet",
      "InterfaceName": "veth0"
    }
  }]
}
EOF

Pull an image and run the container

After you have downloaded and installed firecracker-containerd, you can now pull an image and run the microVM container. The following script starts firecracker-containerd.

firecracker-containerd --config /etc/firecracker-containerd/config.toml

 If you are successful, the terminal will present you with an output similar to the text below.

root@firecracker:/tmp# firecracker-containerd –config /etc/firecracker-containerd/config.toml
INFO[2021-07-09T06:09:51.535598043Z] starting containerd
revision=c2323bc71886b3abfefd9afa53244740f70db0b8 version=1.5.2+unknown
INFO[2021-07-09T06:09:51.724317096Z] loading plugin “io.containerd.content.v1.content”…
type=io.containerd.content.v1
INFO[2021-07-09T06:09:51.724659848Z] loading plugin “io.containerd.snapshotter.v1.
devmapper”… type=io.containerd.snapshotter.v1
INFO[2021-07-09T06:09:51.724787229Z] initializing pool device “fc-dev-thinpool”
INFO[2021-07-09T06:09:51.726961441Z] using dmsetup:
Library version: 1.02.155 (2018-12-18)
Driver version: 4.39.0
INFO[2021-07-09T06:09:52.153939660Z] loading plugin “io.containerd.snapshotter.v1 overlayfs”…  

Now that firecracker-containerd is running, you need to open a new terminal, pull an image, and run the container. The script below pulls and runs the latest Debian image.

 firecracker-ctr --address /run/firecracker-containerd/containerd.sock \\
     image pull \\
     --snapshotter devmapper \\
     docker.io/library/debian:latest

firecracker-ctr --address /run/firecracker-containerd/containerd.sock \\
     run \\
     --snapshotter devmapper \\
     --runtime aws.firecracker \\
     --rm --tty --net-host \\
     docker.io/library/debian:latest \\
     test 

As you can see in the output below, we have successfully started and logged into the microVM.

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> image pull \
> --snapshotter devmapper \
> docker.io/library/debian:latest

docker.io/library/debian:latest: resolved |++++++++++++++++++++++++++++++++++++++| 
index-sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b: done |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:5625c115ad881f19967a9b66416f8d40710bb307ad607d037f8ad8289260f75f: done |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0bc3020d05f1e08b41f1c5d54650a157b1690cde7fedb1fafbc9cda70ee2ec5c: done |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7a4951775d157843b47250a2a5cc7b561d2abe0b29ae6f19737a04635302eacf: done |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.1 s total: 48.1 M (15.4 MiB/s)                    
unpacking linux/amd64 sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b...
done: 3.373106679s

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> run \
> --snapshotter devmapper \
> --runtime aws.firecracker \
> --rm --tty --net-host \
> docker.io/library/debian:latest \
> test
root@microvm:/# 

Scanning a firecracker microVM with MergeBase

Now that we have a working firecracker-containerd image, we can scan it for vulnerabilities with MergeBase. 

The procedure is similar to the one for scanning Docker containers. First, you need to download the mergbase.jar file from the MergeBase portal, as shown in the image below.

Then run the command to scan the image as shown below.

java -jar mergebase.jar --mode=profile --name=firecrackervm debian:buster-slim

As you can see in the output below, MergeBase identified several CVEs

root@firecracker:~# java -jar mergebase.jar --mode=profile --name=
firecrackervm debian:buster-slim

root@firecracker:~# java -jar mergebase.jar --mode=profile --name=
firecrackervm debian:buster-slim
Saving application profile to https://demo.mergebase.com
 
 Vulnerable Files Overview
 =========================
 
 EXTRA-HIGH (1)
 -----
 CVE-2020-6096 (libc-bin@2.28-10.DEBIAN)
 
 HIGH (2)
 -----
 CVE-2021-3326, CVE-2019-9192, CVE-2018-20796 (libc-bin@2.28-10.DEBIAN)
 CVE-2019-9923 (tar@1.30+dfsg-6.DEBIAN)
 
 MEDIUM (3)
 -----
 CVE-2019-5188, CVE-2019-5094 (e2fsprogs@1.44.5-1+deb10u3.DEBIAN)
 CVE-2020-27618, CVE-2019-7309, CVE-2019-25013 (libc-bin@2.28-10.DEBIAN)
 CVE-2021-20193 (tar@1.30+dfsg-6.DEBIAN)
 
 CVEs Overview
 =========================
 EXTRA-HIGH (1)
 -----
 CVE-2020-6096
 
 HIGH (4)
 -----
 CVE-2021-3326, CVE-2019-9923, CVE-2019-9192, CVE-2018-20796
 
 MEDIUM (6)
 -----
 CVE-2021-20193, CVE-2020-27618, CVE-2019-7309, CVE-2019-5188, CVE-2019-5094, CVE-2019-25013
 
Processing time: 5892 ms
 
MergeBase CLT version=v3.0.2 (build-id: bc04183)

If we review the vulnerabilities on the MergeBase portal, the primary issue lies with three Debian components, namely libc-bin, tar, and e2fsprogs.

Conclusion

If we compare the base scans from a Docker image to a Firecracker microVM, the outputs clearly show that microVMs have far fewer vulnerabilities. The previous article illustrated that a base Docker image had 329 vulnerabilities, as shown in the image below.

If we compare the 11 vulnerabilities found on a firecracker microVM, the reduced memory footprint and attack surface area clearly provide a more secure computing environment.

Try it out

Firecracker will really enhance your security through isolation of VM’s and, as we have seen by scanning a firecracker microVM with MergeBase in this article, far fewer vulnerabilities. Try it out yourself by using Firecracker and scanning it with MergeBase.