Webinar Wednesdays – When Dependabot Is Worse Than Nothing: Log4J As A Sub-Dependency

Watch if this applies to you and find out how to fix this :).

from the “Webinar Wednesday  from March 30th, 2022, with Jim and Julius

Why should you care? Because if you’re using industry-standard software leader Dependabot, then your devs didn’t fix the recent Log4J problem properly.
If you’re using Dependabot, then the tools you’re using now aren’t getting the job done.
In practice, Dependabot has a serious implementation flaw: it can only see transitive dependencies (aka sub-dependencies) in languages and dependency managers that support lock files.

In theory Dependabot is exactly what the world needs to keep software dependency chains safe from known vulnerabilities: tightly integrated with Github; auto-generates pull-requests; plugged into Github Security Advisories (GHSA); support for a wide range of programming languages and dependency managers.

But in practice Dependabot has a serious implementation flaw: it can only see transitive dependencies (aka sub-dependencies) in languages and dependency managers that support lock files.

Do you know any languages that currently DO NOT support lock files?

Java / Maven !

This has some bad implications if you’re using Dependabot to protect yourself from Log4J (since Log4J is a Java library).

Webinar Wednesdays – OWASP Top Ten 2021 – Recording

Compromises in the application layer are now responsible for 40% of breaches. Two years ago that was 24%. Obviously time to pay attention to application security. OWASP will give you a running start with their Top Ten

Video recording of OWASP Top Ten 2021 webinar with Jim Manico and Julius Musseau – 45 min on March 16th, 2022

Imagine if a dozen of the top cybersecurity experts in the world reviewed your software for security problems.  Since application security is generally not well covered in university, college, and bootcamp software courses, it’s likely they would probably find a lot of problems!  Of course, hiring even 1 security expert to review your work is out of reach for a lot of software teams – let alone 12 experts. But you can do the next best thing:  you can check out the OWASP Top Ten 2021:

The OWASP Top 10 is an important awareness document for web developers and web application security professionals. It represents a broad industrial consensus from cyber security experts about the most critical security risks to web applications. This webinar provides defensive instruction in relation to the OWASP Top Ten to aid developers towards authoring secure software.

This webinar with Jim Manico and Julius Musseau will cover the OWASP Top-10 (2021 Edition) in depth:

  • A01:2021-Broken Access Control
  • A02:2021-Cryptographic Failure
  • A03:2021-Injection
  • A04:2021-Insecure Design
  • A05:2021-Security Misconfiguration
  • A06:2021-Vulnerable and Outdated Components
  • A07:2021-Identification and Authentication Failures
  • A08:2021-Software and Data Integrity Failures
  • A09:2021-Security Logging and Monitoring Failure
  • A10:2021-Server-Side Request Forgery

About the panelists:

  Jim Manico

  Julius Musseau

Critical Java vulnerability log4j, CVE-2021-44228

Photo Julius Musseau, talking about Critical log4j vulnerability in Java CVE-2021-44228

Log4j is an extremely popular and heavily used open-source Java library that can be used for unauthenticated remote-code execution, making CVE-2021-44228 and CVE-2021-45046 critical vulnerabilities that you must pay attention to.

In other words: if your software uses this library, then there is a good chance that anyone can gain full remote control of your servers and data.

This particular vulnerability is among the worst we’ve seen in the last 5 years, in part because the vulnerability itself is so severe and so easy to trigger. Adding to the severity, this vulnerable library is among the most popular and widely deployed open-source Java libraries on the planet.

The Apache Struts disaster from 2017 (CVE-2017-5638) was a large part of why we launched our startup ( to help companies protect themselves from known-vulnerabilities in open source libraries exactly like these.

This time it’s even worse. Globally, everyone must now urgently scramble to upgrade all applicable software to use Java library log4j version at least 2.15.0 (which was published today). Companies that for whatever reason fail to execute on this upgrade will likely be attacked and exploited within the next few days – assuming they are not already under attack.

We cannot stress how urgent it is for administrators of Java systems to succeed in rolling out this upgrade. Patch this Critical log4j vulnerability in Java CVE-2021-44228 immediately.

There is an easy way to check any server for this vulnerability. At the root of your file system, using an administrator account, download and run our free Log4J-Detector tool:

This command can take several hours to complete. Any occurrence of “_VULNERABLE_” in this command’s output indicates with high probability that you are vulnerable to this severe software exploit.

MergeBase can help you patch your vulnerable systems to eliminate this bug. MergeBase can also help you in situations where patching is not possible. We have a patented cyber-security technique for exactly this situation.

Further reading:

You can try MergeBase to see if you are vulnerable, or download our free tool to detect vulnerable log4j versions on any of your machines and in any application.

You might also be interested in this video on securing log4j instantly with run-time protection.

Scanning a firecracker microVM with MergeBase

Firecracker is a virtual machine monitor that allows you to create and manage microVMs. It leverages the Linux Kernel-based Virtual Machine (KVM) and utilizes a minimalist design for increased security. As firecracker microVMs do not include unnecessary devices and guest functionality, they provide a reduced memory footprint and attack surface area. 

The firecracker architecture is used by and integrated with several infrastructure solutions such as appfleet, containerd,, and OpenNebula. In this article, we will be building a firecracker containerd microVM and scan it for any known vulnerabilities with MergeBase.

Building a firecracker microVM

As firecracker leverages KVM, you need to run it on a bare metal server that supports virtualization, such as an AWS i3.metal instance. You can run it on a virtual machine that utilizes nested virtualization. However, this platform is not supported. The following script tests your current environment and will let you know if it is capable of running firecracker.  

err=""; \
[ "$(uname) $(uname -m)" = "Linux x86_64" ] \
 || err="ERROR: your system is not Linux x86_64."; \
[ -r /dev/kvm ] && [ -w /dev/kvm ] \
 || err="$err\nERROR: /dev/kvm is innaccessible."; \
(( $(uname -r | cut -d. -f1)*1000 + $(uname -r | cut -d. -f2) >= 4014 )) \
 || err="$err\nERROR: your kernel version ($(uname -r)) is too old."; \

dmesg | grep -i "hypervisor detected" \
 && echo "WARNING: you are running in a virtual machine. Firecracker is not well tested under nested virtualization."; \

[ -z "$err" ] && echo "Your system looks ready for Firecracker!" || echo -e "$err"

If your system can run firecracker, the script will confirm it, as illustrated in the output below.

root@firecracker:~# bash
[ 0.000000] Hypervisor detected: KVM

Install the latest version of Go

The firecracker team have created a quick start guide you can use to get your firecracker-containerd instance up and running. However, if you run the script, it will fail as it uses Go modules that leverage later versions of the Go programming language. The workaround for this challenge is to install the latest version of Go manually, as illustrated in the steps below.

Download the latest version of Go

cd /tmp

Extract the tarball to the installation location – /usr/local

tar -C /usr/local -xzf go1.<Version_Number>.linux-amd64.tar.gz

Set the environment

echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.profile
echo "export GOPATH=~/.go" >> ~/.profile
source ~/.profile

 If you have installed the latest version of Go successfully, running the command go version will verify it, as shown in the output below.

root@firecracker:/tmp# go version
go version go1.16.5 linux/amd64

Download and install the required dependencies on a Debian based instance

Now that you have the latest version of Go installed, you can run the script below provided by the firecracker team that downloads and installs the required dependencies on a Debian based instance.

cd ~
# Install git, Go 1.13, make, curl
sudo mkdir -p /etc/apt/sources.list.d
echo "deb buster-backports main" | \
  sudo tee /etc/apt/sources.list.d/buster-backports.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
  install --yes \
  golang-1.13 \
  make \
  git \
  curl \
  e2fsprogs \
  util-linux \
  bc \
# Debian's Go 1.13 package installs "go" command under /usr/lib/go-1.13/bin
export PATH=/usr/lib/go-1.13/bin:$PATH
cd ~
# Install Docker CE
# Docker CE includes containerd, but we need a separate containerd binary, built
# in a later step
curl -fsSL | sudo apt-key add -
apt-key finger | grep '9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88' || echo '**Cannot find Docker key**'
echo "deb [arch=amd64] $(lsb_release -cs) stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
     install --yes \
     docker-ce aufs-tools-
sudo usermod -aG docker $(whoami)
# Install device-mapper
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y dmsetup

Download and install firecracker-containerd

Once you have downloaded and installed the dependencies, you can move on to the next step and get firecracker-containerd downloaded and installed. Again, we will use the script below provided by the firecracker team.

cd ~
# Check out firecracker-containerd and build it. This includes:
# * firecracker-containerd runtime, a containerd v2 runtime
# * firecracker-containerd agent, an inside-VM component
# * runc, to run containers inside the VM
# * a Debian-based root filesystem configured as read-only with a read-write
# overlay
# * firecracker-containerd, an alternative containerd binary that includes the
# firecracker VM lifecycle plugin and API
# * tc-redirect-tap and other CNI dependencies that enable VMs to start with
# access to networks available on the host
git clone
cd firecracker-containerd
sg docker -c 'make all image firecracker'
sudo make install install-firecracker demo-network
cd ~
# Download kernel
curl -fsSL -o hello-vmlinux.bin
# Configure our firecracker-containerd binary to use our new snapshotter and
# separate storage from the default containerd binary
sudo mkdir -p /etc/firecracker-containerd
sudo mkdir -p /var/lib/firecracker-containerd/containerd
# Create the shim base directory for which firecracker-containerd will run the
# shim from
sudo mkdir -p /var/lib/firecracker-containerd
sudo tee /etc/firecracker-containerd/config.toml <<EOF
disabled_plugins = ["cri"]
root = "/var/lib/firecracker-containerd/containerd"
state = "/run/firecracker-containerd"
  address = "/run/firecracker-containerd/containerd.sock"
    pool_name = "fc-dev-thinpool"
    base_image_size = "10GB"
    root_path = "/var/lib/firecracker-containerd/snapshotter/devmapper"
  level = "debug"
# Setup device mapper thin pool
sudo mkdir -p /var/lib/firecracker-containerd/snapshotter/devmapper
cd /var/lib/firecracker-containerd/snapshotter/devmapper
if [[ ! -f "${DIR}/data" ]]; then
    sudo touch "${DIR}/data"
    sudo truncate -s 100G "${DIR}/data"
if [[ ! -f "${DIR}/metadata" ]]; then
    sudo touch "${DIR}/metadata"
    sudo truncate -s 2G "${DIR}/metadata"
DATADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/data)"
if [[ -z "${DATADEV}" ]]; then
    DATADEV="$(sudo losetup --find --show ${DIR}/data)"
METADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/metadata)"
if [[ -z "${METADEV}" ]]; then
    METADEV="$(sudo losetup --find --show ${DIR}/metadata)"
DATASIZE="$(sudo blockdev --getsize64 -q ${DATADEV})"
echo "${THINP_TABLE}"
if ! $(sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
    sudo dmsetup create "${POOL}" --table "${THINP_TABLE}"
cd ~
# Configure the aws.firecracker runtime
# The long kernel command-line configures systemd inside the Debian-based image
# and uses a special init process to create a read-write overlay on top of the
# read-only image.
sudo mkdir -p /var/lib/firecracker-containerd/runtime
sudo cp ~/firecracker-containerd/tools/image-builder/rootfs.img /var/lib/firecracker-containerd/runtime/default-rootfs.img
sudo cp ~/hello-vmlinux.bin /var/lib/firecracker-containerd/runtime/default-vmlinux.bin
sudo mkdir -p /etc/containerd
sudo tee /etc/containerd/firecracker-runtime.json <<EOF
  "firecracker_binary_path": "/usr/local/bin/firecracker",
  "cpu_template": "T2",
  "log_fifo": "fc-logs.fifo",
  "log_levels": ["debug"],
  "metrics_fifo": "fc-metrics.fifo",
  "kernel_args": "console=ttyS0 noapic reboot=k panic=1 pci=off nomodules ro systemd.journald.forward_to_console init=/sbin/overlay-init",
  "default_network_interfaces": [{
    "CNIConfig": {
      "NetworkName": "fcnet",
      "InterfaceName": "veth0"

Pull an image and run the container

After you have downloaded and installed firecracker-containerd, you can now pull an image and run the microVM container. The following script starts firecracker-containerd.

firecracker-containerd --config /etc/firecracker-containerd/config.toml

 If you are successful, the terminal will present you with an output similar to the text below.

root@firecracker:/tmp# firecracker-containerd --config /etc/firecracker-containerd/config.toml
INFO[2021-07-09T06:09:51.535598043Z] starting containerd revision=c2323bc71886b3abfefd9afa53244740f70db0b8 version=1.5.2+unknown
INFO[2021-07-09T06:09:51.724317096Z] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
INFO[2021-07-09T06:09:51.724659848Z] loading plugin "io.containerd.snapshotter.v1.devmapper"... type=io.containerd.snapshotter.v1
INFO[2021-07-09T06:09:51.724787229Z] initializing pool device "fc-dev-thinpool"  
INFO[2021-07-09T06:09:51.726961441Z] using dmsetup:
Library version: 1.02.155 (2018-12-18)
Driver version: 4.39.0 
INFO[2021-07-09T06:09:52.153939660Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"... 

Now that firecracker-containerd is running; you need to open a new terminal, pull an image, and run the container. The script below pulls and runs the latest Debian image.

firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
     image pull \
     --snapshotter devmapper \
firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
     run \
     --snapshotter devmapper \
     --runtime aws.firecracker \
     --rm --tty --net-host \ \

As you can see in the output below, we have successfully started and logged into the microVM.

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> image pull \
> --snapshotter devmapper \
> resolved |++++++++++++++++++++++++++++++++++++++| 
index-sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b: done |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:5625c115ad881f19967a9b66416f8d40710bb307ad607d037f8ad8289260f75f: done |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0bc3020d05f1e08b41f1c5d54650a157b1690cde7fedb1fafbc9cda70ee2ec5c: done |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7a4951775d157843b47250a2a5cc7b561d2abe0b29ae6f19737a04635302eacf: done |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.1 s total: 48.1 M (15.4 MiB/s)                    
unpacking linux/amd64 sha256:bd937dbe9aba256821659d81b49f70c58da0b2e042d72a8860a0f72c17b5a84b...
done: 3.373106679s

root@firecracker:~# firecracker-ctr --address /run/firecracker-containerd/containerd.sock \
> run \
> --snapshotter devmapper \
> --runtime aws.firecracker \
> --rm --tty --net-host \
> \
> test

Scanning a firecracker microVM with MergeBase

Now that we have a working firecracker-containerd image, we can scan it for vulnerabilities with MergeBase. T

The procedure is similar to the one for scanning Docker containers. First, you need to download the mergbase.jar file from the MergeBase portal, as shown in the image below.

 Then run the command to scan the image as shown below.

java -jar mergebase.jar --mode=profile --name=firecrackervm debian:buster-slim

As you can see in the output below, MergeBase identified several CVEs

root@firecracker:~# java -jar mergebase.jar --mode=profile --name=firecrackervm debian:buster-slim
Saving application profile to
 Vulnerable Files Overview
 CVE-2020-6096 (libc-bin@2.28-10.DEBIAN)
 HIGH (2)
 CVE-2021-3326, CVE-2019-9192, CVE-2018-20796 (libc-bin@2.28-10.DEBIAN)
 CVE-2019-9923 (tar@1.30+dfsg-6.DEBIAN)
 CVE-2019-5188, CVE-2019-5094 (e2fsprogs@1.44.5-1+deb10u3.DEBIAN)
 CVE-2020-27618, CVE-2019-7309, CVE-2019-25013 (libc-bin@2.28-10.DEBIAN)
 CVE-2021-20193 (tar@1.30+dfsg-6.DEBIAN)
 CVEs Overview
 HIGH (4)
 CVE-2021-3326, CVE-2019-9923, CVE-2019-9192, CVE-2018-20796
 CVE-2021-20193, CVE-2020-27618, CVE-2019-7309, CVE-2019-5188, CVE-2019-5094, CVE-2019-25013
Processing time: 5892 ms
MergeBase CLT version=v3.0.2 (build-id: bc04183)

 If we review the vulnerabilities on the MergeBase portal, the primary issue lies with three Debian components, namely libc-bin, tar, and e2fsprogs.


If we compare the base scans from a Docker image to a Firecracker microVM, the outputs clearly show that microVMs have far fewer vulnerabilities. The previous article illustrated that a base Docker image had 329 vulnerabilities, as shown in the image below.


If we compare the 11 vulnerabilities found on a microVM, the reduced memory footprint and attack surface area clearly provide a more secure computing environment.

Try it out

Firecracker will really enhance your security through isolation of VM’s and, as we have seen by scanning a firecracker microVM with MergeBase in this article, far fewer vulnerabilities. Try it out yourself by using Firecracker and scanning it with MergeBase.

Ubuntu Docker Container Scanning – Is it enough to protect your app?

In this post, we are going to look at Docker container scanning:

  1. First, we will build an Ubuntu Docker container, add some software to it, run a Docker scan, and record the vulnerabilities.
  2. Then, we will try to resolve some of those vulnerabilities, run another scan, and examine the results.
  3. Finally, we will determine whether container scanning is enough to protect your applications and data from compromise. 

Building the Docker Container

We will build our Ubuntu Docker container using the Ubuntu Xenial image. Since it is a few versions behind, it will hopefully contain a few vulnerabilities. We are also going to install Python, PIP3, and a vulnerable version of urllib3. If we search the CVE database, we find CVE-2021-28363. According to this CVE, the urllib3 library 1.26.x before 1.26.4 omits SSL certificate validation in some cases involving HTTPS to HTTPS proxies.

The first step in our process is to pull the Ubuntu Xenial Docker image from Docker Hub and run it as a container on our Docker host. The command below will get the image, start the container in interactive mode, and name it ubuntu_xenial. 

docker run -it –name ubuntu_xenial ubuntu:xenial

Once the process completes, you will be presented with the Ubuntu prompt, as shown in the text output below.

PS C:\Users\docker> docker run -it --name ubuntu_xenial ubuntu:xenial

Unable to find image 'ubuntu:xenial' locally

xenial: Pulling from library/ubuntu

80bce60046fa: Pull complete

55a738a15540: Pull complete

e19cf0706c62: Pull complete

de4cdd6c27d1: Pull complete

Digest: sha256:9775877f420d453ef790e6832d77630a49b32a92b7bedf330adf4d8669f6600e

Status: Downloaded newer image for ubuntu:xenial


The next step is to install Python and PIP by running the following command:

apt update && apt install python3-pip

root@a317929cf5d7:/# apt update && apt install python3-pip

Get:1 xenial InRelease [247 kB]

Get:2 xenial-security InRelease [109 kB]

Get:3 xenial-updates InRelease [109 kB]

Get:4 xenial-security/main amd64 Packages [2051 kB]
Fetched 19.4 MB in 42s (458 kB/s)

Reading package lists... Done

Building dependency tree

Reading state information... Done

5 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done
The following packages will be upgraded:


1 upgraded, 85 newly installed, 0 to remove and 4 not upgraded.

Need to get 98.7 MB of archives.

After this operation, 289 MB of additional disk space will be used.

Do you want to continue? [Y/n]


After installing Python and PIP, we are ready to install urllib3. However, since the development team fixed the vulnerability in version 1.26.4, we need to install an earlier vulnerable version. For the purposes of this exercise, let’s install version 1.26.0 from requirements.txt. 

We can check that our requirements.txt file includes our vulnerable version by running the cat command.

root@a317929cf5d7:/# cat requirements.txt


We can use PIP to install the  requirements.txt file.

pip3 install -r requirements.txt

root@a317929cf5d7:/# pip3 -r requirements.txt

Collecting urllib3==1.26.0

  Downloading (136kB)

    100% |################################| 143kB 3.9MB/s

Installing collected packages: urllib3

Successfully installed urllib3-1.26.0

You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

root@a317929cf5d7:/urllib3_test# pip3 freeze requirements.txt


You are using pip version 8.1.1, however version 21.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

Now that we have updated our container, it is time to commit it and create our image by running the following Docker command from the docker host.

docker commit ubuntu_xenial

PS C:\Users\docker> docker commit ubuntu_xenial


Let’s check that Docker created the image by running the docker images command.

docker images

As we can see in the text output below, Docker created the image. 

PS C:\Users\docker> docker images


<none>      <none>    130de6a9e382   About a minute ago   465MB

ubuntu       xenial   9ff95a467e45   12 days ago          135MB

However, to make our lives easier, let’s give the new image a tag to identify it by running the following command.

docker tag 130de6a9e382 our_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag 130de6a9e382 our_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED        SIZE

our_ubuntu_image  latest   130de6a9e382   3 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago     135MB

As Docker’s scanning feature only works on images and not containers, we are now ready to scan the image for any vulnerabilities. Let’s start the Docker container scanning by running the following command.

docker scan our_ubuntu_image

PS C:\Users\docker> docker scan our_ubuntu_image

Docker Scan relies upon access to Snyk, a third party provider, do you consent to proceed using Snyk? (y/N)docker y

/ Analyzing container dependencies for our_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 193 issues, as shown in the text output below.

Package manager:  deb

Project name:     docker-image|our_ubuntu_image

Docker image:     our_ubuntu_image

Platform:         linux/amd64

Licenses:        enabled

Tested 185 dependencies for known issues, found 193 issues.

Vulnerability Analysis and Resolution

If we study the vulnerability report, we find that Docker container scanning highlights some vulnerabilities that can be fixed, as shown in the text output below.

✗  Low severity vulnerability found in glibc/libc-bin

  Description: Improper Data Handling


  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

✗ Low severity vulnerability found in glibc/libc-bin

  Description: Integer Underflow


  Introduced through: glibc/libc-bin@2.23-0ubuntu11.2, meta-common-packages@meta

  From: glibc/libc-bin@2.23-0ubuntu11.2

  From: meta-common-packages@meta > glibc/multiarch-support@2.23-0ubuntu11.2

  Fixed in: 2.23-0ubuntu11.3

If we look at the message for the two vulnerabilities in the image, we need to update those particular packages to resolve the issue. First, we need to create a container from the new image to make sure we are working with the latest version. As we did previously, we need to execute the docker run command as shown in the image below.

docker run -it –name our_ubuntu_image our_ubuntu_image

PS C:\Users\docker> docker run -it --name our_ubuntu_image our_ubuntu_image


After executing the docker run command, we can run the following commands to update Ubuntu from the container’s command line.

apt update && apt upgrade

root@0dfcbff2e3d1:/# apt update && apt upgrade

Hit:1 xenial-security InRelease

Hit:2 xenial InRelease

Hit:3 xenial-updates InRelease

Hit:4 xenial-backports InRelease

Reading package lists... Done

Building dependency tree

Reading state information... Done

4 packages can be upgraded. Run 'apt list --upgradable' to see them.

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

The following packages will be upgraded:

  apt libapt-pkg5.0 libc-bin multiarch-support

4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Need to get 2458 kB of archives.

After this operation, 70.7 kB of additional disk space will be used.

Do you want to continue? [Y/n]

After running and installing the updates, we need to commit and create another Docker image to save the changes. 

docker commit our_ubuntu_image

PS C:\Users\docker> docker commit our_ubuntu_image


We can check that Docker created the image successfully by running the docker images command.

docker images

As we can see in the text output below, Docker confirms the new updated image exists.

PS C:\Users\docker> docker images

REPOSITORY         TAG       IMAGE ID       CREATED          SIZE

<none>             <none>    bf5809ae94d9   2 minutes ago    476MB

our_ubuntu_image   latest    130de6a9e382   19 minutes ago   465MB

ubuntu             xenial    9ff95a467e45   12 days ago      135MB

As we did before, let’s give the new image a tag to identify it by running the following command.

docker tag bf5809ae94d9 updated_ubuntu_image

If we rerun the docker images command, we can see that Docker updated the information.

PS C:\Users\docker> docker tag bf5809ae94d9 updated_ubuntu_image

PS C:\Users\docker> docker images

REPOSITORY             TAG       IMAGE ID       CREATED          SIZE

updated_ubuntu_image   latest    bf5809ae94d9   3 minutes ago   476MB

our_ubuntu_image       latest    130de6a9e382   21 minutes ago 465MB

ubuntu                 xenial    9ff95a467e45   12 days ago       135MB

We are now ready to scan the updated container to see if our efforts resolved any vulnerabilities. As before, to run docker scan execute the following command. Note that we are now scanning our updated image.

docker scan updated_ubuntu_image

PS C:\Users\docker> docker scan updated_ubuntu_image

/ Analyzing container dependencies for updated_ubuntu_image

/ Querying vulnerabilities database...

Once the scan completes, the final report indicates 191 issues, as shown in the text output below. This result demonstrates that updating our Ubuntu container resolved two vulnerabilities. 

Package manager:    deb

Project name:      docker-image|updated_ubuntu_image

Docker image:      updated_ubuntu_image

Platform:          linux/amd64

Licenses:         enabled

Tested 185 dependencies for known issues, found 191 issues.

However, if we search both the original vulnerability report and the report generated after our updates, neither report contains an alert for CVE-2021-28363. If you recall, when we created our image, we installed a vulnerable version of urllib3. Since neither scan picked up this vulnerability, we need to consider alternatives that mitigate this risk.

MergeBase Vulnerability Analysis and Resolution

To test the validity of Docker scan’s analysis and reporting, let’s run the same scans using MergeBase.

As in the first example, we first need to run a scan on the Docker image named our_ubuntu_image before we run any Ubuntu updates. 

First, we need to get the docker image repository and tag by running the following command:

docker images

As we can see in the text output below, the repository name is ‘our_ubuntu_image,’ and its tag is ‘latest.’


our_ubuntu_image latest 130de6a9e382 18 minutes ago 462MB

ubuntu xenial 9ff95a467e45 3 weeks ago 135MB

Using the MergeBase command-line tool, we can scan this image using the following command:

java -jar mergebase.jar –mode=profile –name=our_ubuntu_image our_ubuntu_image:latest

If we look at the summary scan report on the MergeBase portal, we can see that it found 301 vulnerabilities. This number exceeds the 193 found by Docker scan by 108 vulnerabilities.

Below is the text output from the command-line tool. As you can see, it details multiple CVEs and categorizes them by severity.

java -jar mergebase.jar --mode=profile --name=updated_ubuntu_image updated_ubuntu_image:latest

Saving application profile to

  Vulnerable Files Overview




  CVE-2017-7614, CVE-2017-7226, CVE-2017-6969 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-8283 (dpkg-dev@1.18.4ubuntu1.7.UBUNTU)

  CVE-2017-8283 (dpkg@1.18.4ubuntu1.7.UBUNTU)

  CVE-2019-18218 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2008-1530 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2016-6309 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2017-12814 (perl-base@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2017-12814 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1126 (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-15801, CVE-2017-1000158 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2018-20839, CVE-2018-15688, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)



  CVE-2019-3462 (apt@1.2.35.UBUNTU)

  CVE-2016-7543 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2020-6096 (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-6096 (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2019-11922 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2017-17522 (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2017-17522 (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  HIGH (18)


  CVE-2018-6557 (base-files@9.4ubuntu4.13.UBUNTU)

  CVE-2019-18276, CVE-2017-5932, CVE-2016-0634 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3549, CVE-2021-20294, 73 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2015-8865, CVE-2014-9653 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2021-3345, CVE-2019-14855, 2 more... (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2018-5733, CVE-2018-5732, CVE-2017-3144 (isc-dhcp-client@4.3.3-5ubuntu12.10.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-3326, CVE-2019-9192, 4 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2016-6305 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2018-6952, CVE-2018-6951, CVE-2018-20969 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2016-2381 (perl@5.22.1-9ubuntu0.9.UBUNTU)

  CVE-2018-1125, CVE-2018-1124, 2 more... (procps@3.3.10-4ubuntu2.5.UBUNTU)

  CVE-2019-20916 (python3-pip@8.1.1-2ubuntu0.6.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-26116, CVE-2019-16056, 3 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-1712, CVE-2019-3844, 8 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2016-6321 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2018-7738 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  MEDIUM (18)


  CVE-2018-0501, CVE-2016-1252 (apt@1.2.35.UBUNTU)

  CVE-2016-9401 (bash@4.3-14ubuntu1.4.UBUNTU)

  CVE-2021-3487, CVE-2021-20284, 80 more... (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2017-18018 (coreutils@8.25-2ubuntu3~16.04.UBUNTU)

  CVE-2017-1000249, CVE-2014-9621, CVE-2014-9620 (file@5.25-2ubuntu1.4.UBUNTU)

  CVE-2011-2207 (gnupg@1.4.20-1ubuntu3.3.UBUNTU)

  CVE-2019-20795 (iproute2@4.3.0-1ubuntu3.16.04.5.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2020-27618, CVE-2019-7309, 7 more... (libc-dev-bin@2.23-0ubuntu11.3.UBUNTU)

  CVE-2021-24032, CVE-2021-24031 (libzstd1@1.3.1+dfsg-1~ubuntu0.16.04.1.UBUNTU)

  CVE-2018-0739, CVE-2016-6308, CVE-2016-6307 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  CVE-2019-20633, CVE-2016-10713 (patch@2.7.5-1ubuntu0.16.04.2.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5-minimal@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2021-3426, CVE-2020-8492, 4 more... (python3.5@3.5.2-2ubuntu0~16.04.13.UBUNTU)

  CVE-2020-13776, CVE-2020-13529, 3 more... (systemd@229-4ubuntu21.31.UBUNTU)

  CVE-2021-20193 (tar@1.28-2.1ubuntu0.2.UBUNTU)

  CVE-2016-5011 (util-linux@2.27.1-6ubuntu3.10.UBUNTU)

  CVE-2021-28363 requirements.txt (urllib3@1.26.0)

  LOW (4)


  USN-3156-0002 (apt@1.2.35.UBUNTU)

  CVE-2020-35448 (binutils@2.26.1-1ubuntu1~16.04.8.UBUNTU)

  CVE-2019-1552 (openssl@1.0.2g-1ubuntu4.19.UBUNTU)

  USN-4120-0002, CVE-2019-20386, 2 more... (systemd@229-4ubuntu21.31.UBUNTU)

Included in the list of vulnerabilities identified by MergeBase is the vulnerable version of urllib3 we installed.

To ensure we are comparing like for like, we should also run the MergeBase scan on the updated image after applying the Ubuntu updates as we did with Docker scan. We can reuse the same MergeBase command line after updating the relevant parameters.

java -jar mergebase.jar –mode=profile –name=updated_ubuntu_image updated_ubuntu_image:latest

Interestingly, the results for the updated Ubuntu image are identical before and after running the updates. 

We can deduct the following outcomes from this exercise:

  1. Docker Scan and MergeBase find different vulnerabilities on the same containers.
  2. MergeBase provides far more detailed results when it comes to the actual software components installed on a container. The text output from the command line tool and the report it generates even details the relevant CVEs. 
  3. The Docker scan results change after running system updates while Mergebase remains the same. This difference indicates that Docker scan focuses more on the operating system while MergeBase concentrates on its specialty, Software Composition Analysis.

Container Scanning Limitations

Our rudimentary experiment shows that default container scanning solutions do not find application-layer vulnerabilities. Consequently, even though these tools provide a critical service that helps you find security issues in your containerized application, they do not offer a complete solution. For example, if we build a Python app and use the vulnerable urllib3 library, our containerized app is not secure. In that instance, a container scan will not show any vulnerabilities, but an attacker could create a rogue certificate authority and compromise your containerized service.

However, this simple illustration is only one example. If we consider that researchers found 18,325 vulnerabilities in 2020 and 7,545 in the first five months of 2021, protecting your containerized application requires multiple layers of security. If you only rely on container scanning solutions, you are not getting the visibility you need for application-layer vulnerabilities, as our urllib3 example demonstrates. What you need is a layered defense-in-depth security approach. 

Defense in Depth for Containerized Applications

While containerized applications offer several benefits such as application consistency, scalability, and cost-effectiveness, managing all the moving parts that provide the platform running your app can be challenging. Since hosts, images, and containers make up a standard Docker architecture, a vulnerability in any of these three critical components could lead to a security incident. 

For example, should a threat actor manage to compromise an image on Docker Hub, it could lead to a devastating supply chain attack. A vulnerability in your Docker host, whether you use a cloud platform, Windows, or Linux server, could also lead to a compromise of your container. Finally, there is the container itself. As we have illustrated with our simple example, standard container scanning apps do not find application-layer vulnerabilities. 

Implementing defense-in-depth security for your containerized application requires a holistic approach. First, you need to secure your platform from external threats by implementing the relevant network segmentation, firewalls, and similar technologies that prevent unauthorized access. Then you need to continuously harden your hosts, images, and containers by leveraging Docker container scanning and implementing patches and upgrades. 

The final element you need to secure in your defense-in-depth strategy is your application layer. Since firewalls, and vulnerability scanners that focus on hosts, images, and containers, do not identify any application-related security threats, you need a solution to identify and mitigate this risk. As our example illustrated, Software Composition Analysis (SCA) tools like MergeBase, provide visibility into the real risk of enterprise applications. Since most, if not all, modern applications leverage external libraries and packages, identifying vulnerabilities in these elements is vital in securing your application. If we revisit our urllib3 example, scanning the containerized application with an SCA tool like MergeBase found the vulnerability and 107 others the Docker scan failed to identify.

Software Composition Analysis (SCA) vs. Java Über Jars

Image source:


Über jars are a type of reuseable Java library that applications sometimes (knowingly or not) incorporate into their systems. Über jars are particularly challenging for software composition analysis (SCA) tools to understand because their structure and organization is complex. In this blog post I explain what über jars are, why they exist, and I provide a mini-benchmark to see how current SCA tools deal with this type of Java library.


Circa February 2021, both MergeBase and Sonatype use deep binary analysis to measure software composition within über jars.

  • MergeBase is the only SCA tool I observed with comprehensive support for über jars.
  • Sonatype also provides decent support but misses a few obvious cases. (Strangely, their legal analysis completely ignores their über jar results.)
  • OWASP Depedency-Check and JFrog Xray are not bad, but their scanning is based solely on metadata files found inside über jars.
  • Snyk, WhiteSource, and Github Dependabot currently have no ability to understand über jars at all.

I did not benchmark Black Duck (I don’t have access to that tool).


Über jars are the Java equivalent of taking everything in your fridge and throwing it all into your largest pot, giving it a good stir. From an SCA (Software Composition Analysis) perspective they are a bit of a nightmare.

Recall: normally you point your SCA tool at a Java jar file (a reuseable Java library) and your SCA tool responds by telling you the jar file’s name, version, and known-vulnerabilities.

But what if that single Java jar file is actually an agglomeration of dozens of Java jar files? What if some maniac cracked open all your jar files and poured all their contents into a single mega jar? That’s exactly what an über jar is and your SCA tool is going to need to reverse-engineer the contents accurately before it can say anything.

Why would anyone create an über jar in the first place?

Java programs are awkward to invoke. You have to tell Java the exact locations of all the jar files your program is using. Über jars are a way around that problem.

For example, a typical normal java program is started like this:

java -classpath lib1.jar:lib2.jar:main.jar name.of.MainEntry

With an über jar it’s less typing, since “lib1.jar” and “lib2.jar” have been blended directly into a single “main-uber.jar” file:

java -classpath main-uber.jar name.of.MainEntry

In this way über jars make Java programs easier to distribute and easier to start. That’s the main reason why they exist.

Recall that Jar files are actually just zip files. You can rename them to “.zip” and then double-click on them if you ever want to see what’s inside them. Über jars are what you get if you unzipped all of your Jar files and combined all the contents into a single zip file instead.

Über jars are a challenge for SCA

Most SCA tools are geared towards providing a single succinct answer for each library they scan.

SCA Scan Results

Identified Library:
Apache Commons-Collections 3.2.1.

CVE-2017-15708, CVE-2015-7501, and CVE-2015-6420

With über jars the answer is more complicated. “Well, actually… this library is a combination of many libraries.”

At MergeBase we analyze every jar file against our master database for this possibility. For example, consider “apacheds-all-1.5.5.jar”, a large über jar containing over 500,000 lines of code coming from dozens of libraries. When we compare this jar file against all known versions of “slf4j-api” here are the results:

Match RatioKnown Library Version
These results show that version 1.5.6 of slf4j-api is contained inside the apacheds-all-1.5.5 über jar file.

In the “slf4j-api” case there is also another hint inside the über jar. If I grep the jar’s contents for “sl4fj-api” I see these two entries:


Opening the latter, I see this:

    #Generated by Maven
    #Fri Nov 21 14:48:07 CET 2008

This gives me further confidence that my binary analysis is correct: version 1.5.6 aligns with my MergeBase result. Some SCA scanners only consider this metadata when examining über jars, but philosophically I don’t agree with that approach, since metadata is not always present, as in the bouncy-castle example below. Metadata is also vulnerable to transcription mistakes and tampering.

You might be curious why this metadata is even present in the first place.

My own theory: it was probably present in the original “slf4j-api” jar. Über jars don’t just combine the software files – they combine all the files! And so if a metadata file is present in the original “slf4j-api” file, it will be dutifully copied into the über jar. I can download the original and see for myself. Sure enough, running “unzip -l slf4j-api-1.5.6.jar” shows both of those metadata files were in the original.

Moving onto to an example without metadata, here’s the results when we compare our über jar against “bcprov-jdk15”:

Match RatioKnown Library Version

There is no metadata available to warn consumers that the highly vulnerable version 1.40 of bcprov-jdk15 was copied into apacheds-all-1.5.5.jar. Unfortunately bcprov-jdk15@1.40 contains over 15 known-vulnerabilities. Scanners that rely on metadata (such as JFrog Xray and OWASP Dependency-Check) will miss this. And of course scanners that lack über jar handling (such as WhiteSource and Snyk) will also miss this.

Using our high-confidence matches we then query our known-vulnerability database for any corresponding vulnerabilities. Our technique is based on binary analysis – no metadata is involved at all, since metadata can be inaccurate. Using this technique we are able to identify dozens of sub-components encapsulated by the apacheds-all-1.5.5 über jar. Here’s a partial listing based on MergeBase’s analysis:

  1. 100.0% – antlr/antlr@2.7.7
  2. 100.0% – commons-io/commons-io@1.4
  3. 100.0% – commons-lang/commons-lang@2.4
  4. 100.0% –
  5. 100.0% –
  6. 100.0% – org.apache.mina/mina-core@2.0.0-M6
  7. 100.0% – org.bouncycastle/bcprov-jdk15@1.43
  8. 100.0% – org.slf4j/slf4j-api@1.5.8

(Etc… 25 more sub-components identified!)

Quick Competitive Check

We were curious to see if competing SCA tools are able to handle über jars. What follows is a quick benchmark against a half-dozen popular SCA tools.


For each SCA tool (MergeBase, OWASP Dependency-Check, Snyk, WhiteSource, Sonatype, etc…):

  1. Git clone:
  2. Run “mvn install”.
  3. Apply each SCA tool against the built “vuln-example-apacheds-all”.
  4. Observe and compare the scan results.

Mini-Benchmark Results

As of February 2021, the apacheds-all-1.5.5 über jar contains two vulnerable sub-components. One of these (bcprov-jdk15@1.40) can only be identified using binary approaches since it had no metadata in the first place, and one of these (commons-collections@3.2.1) can be identified either via binary approaches or via metadata scanning.

We group the benchmark results into 3 categories:

1. Scanners that do not support über jars at all.

Snyk and Whitesource appear to have no idea that “apacheds-all@1.5.5” is made by combining many jar files together.

Similarly, Github’s Dependabot also has no idea about this.

2. Scanners that support a metadata-based understanding of über jars.

OWASP Dependency-Check and JFrog Xray both detect the “commons-collections@3.2.1” metadata inside the über jar.

3. Scanners that support deep understanding of über jars.

Sonatype fails to identify any known-vulnerabilities with respect to commons-collections@3.2.1, and yet it does correctly identify that apacheds-all@1.5.5 contains bcprov-jdk15@1.40! This is a lopsided result: Sonatype clearly has a deep understanding here (otherwise it would be impossible to identify bcprov-jdk15), and yet somehow Sonatype is failing to spot the easy one. We also noted that Sonatype reported the license as Apache 2.0, when bcprov-jdk15 uses the MIT license.

MergeBase identifies all vulnerabilities correctly in this case. 🙂


Über jars are a special type of Java software component made by combining several jars into a single jar. Aside from MergeBase, most SCA scanners currently provide sub-par or even zero support for this component type.

Last piece of advice: Have Über jars? Give MergeBase a closer look!

Special Thanks

Specials thank to Dr. Ken Warkentyne, our principal engineer, who built MergeBase’s über jar scanning capabiliity.

Scanning .NET and Nuget projects for known vulnerabilities

We recently (August 2020) completed version 1 of our .NET scanner. Its goal is scanning .NET and Nuget projects for libraries with known vulnerabilities in any .NET project.

For this blog post we thought we’d take our scanner out for a spin and see how it compares against the competition.


Here are the results of scanning .NET and Nuget projects for known vulnerabilities:

  • MergeBase – 18 vulnerabilities, 0 false positives.
  • Snyk – 7 vulns and 5 false, or 4 vulns and 0 false (depends on scanner setup).
  • WhiteSource – 12 vulns, 0 false.
  • OWASP Dependency Check – 12 vulns, 17 false.
  • Dotnet Retire – 2 vulns, 0 false.
  • Sonatype – 0 vulns, 0 false.
  • Dependabot – 0 vulns, 0 false.


We chose the .NET Orleans project as to scan for .NET and Nuget vulnerabilities. It’s active, complex, and it builds successfully (August 6th, 2020, master = 2e10856f7b7ed9443c). We also liked how this project contained a mix of Nuget styles (e.g., older “packages.config” style as well as the newer “<PackageReference/>” style).

We type “dotnet build” before scanning. This way .NET scanners can use the generated “obj/project.assets.json” files to supplement their scan data if they want to, and “dotnet build” is such a critical step for building any .NET project that we think it’s safe for an SCA tool to assume this command has completed successfully.

As for comparing results, we count CVE’s. If the scan outputs 1 or 300 or 9,000,000 hits against CVE-2018-8292, we count that as a single CVE. We then do a quick “desk check” to categorize the result as either a true-hit, a false-negative, or an ambiguous result (where it’s hard to say one way or the other). The “desk check” is very much based on my own decades of experience as a software engineer – I encourage others to rerun these scans and see if they agree or disagree.

Because this is a .NET scan for Nuget project, we ignore any results the scanners find from other file-types lying on the file system (e.g., “VotingWeb/wwwroot/lib/jquery/jquery.min.js”). We do, however, count results found from nuget references into other language artifacts (e.g., “GPSTracker.Web/packages.config” contains a nuget reference to “<package id=”bootstrap” version=”3.0.0″ targetFramework=”net45″ />” in its packages.config file – we’ll count this.)

Here is the exact sequence of steps:

  1. git clone
  2. dotnet build
  3. Deploy The Scanners!
  4. Validate the results.

A note about ambiguous results:

We classify some results as ambiguous. This means there’s definitely some smoke, so we can’t rule immediately it out as a false negative after examining the metadata, but on the other hand, there’s enough uncertainty to also make us uncomfortable considering it a true hit.


The vulnerability references “bootstrap” in the scan report but the CVE description talks about “bootstrap-sass”. Maybe? Or in another case the CVE description starts out with the words (in all caps) “DISPUTED”.

Results of scanning .NET, Nuget project for vulnerabilities:

I’ll save the best for first! Here’s what MergeBase finds:

1. MergeBase

18 vulnerabilities found (and two ambiguous hits).

Drop the scanner into the Orleans subdirectory. Type “java -jar mergebase.jar .” and the results are pretty straightforward: 2 critical CVE’s, 5 high ones, and 11 mediums. A quick spot-check of the metadata looked good (no false positives and two ambiguous results).

2. Github Dependabot

Zero vulnerabilities found.

Dependabot not doing too much here, despite being a Microsoft product (albeit recently acquired):

3. Dotnet Retire

Two vulnerabilities found: CVE-2018-8292 and CVE-2018-8416. MergeBase also found these two among the 18 vulnerabilities it identified.

4. Sonatype

Zero .NET vulnerabilities found!

Sonatype does detect a small handful of JavaScript vulnerabilities (since Orleans contains things like “VotingWeb/wwwroot/lib/jquery/jquery.min.js”), but nothing for .NET. To be fair, their scanner instructions did say “you must copy all .NET packages you depend on into the zip file you are scanning beforehand.” I typed “dotnet build” and zipped the result (660MB). As far as I’m concerned, I was doing them a favour by even zipping up orleans post-build in the first place – no other scanner required that.

Note-to-self: Probably MergeBase should also scan those JavaScript packages! (Our current logic looks for NPM and Yarn lock files, but maybe it’s time to roll up our sleeves and consider scanning raw *.js and *.min.js files, too.)

5. Snyk

“It’s Complicated!” The problem with Snyk is that there’s two different ways to invoke the Snyk scanner, and each way returns wildly different results.

Snyk Approach #1 – Github Integration:

15 vulnerabilities found. 5 of those are false positives (all because Microsoft.NETCore.App was flagged as a dependency, but it’s not). 3 are ambiguous. 7 are true hits.

A few NPM and Docker vulnerabilities also found, but seeing as this bakeoff is only about .NET we ignored those.

Snyk Approach #2 – Command Line Invocation:

7 vulnerabilities found. 3 are ambiguous, leaving 4 true hits, including 1 true hit that Snyk approach #1 above did not find (CVE-2020-1469).

No NPM or Docker vulnerabilities found via this approach.

6. Whitesource Bolt

12 true CVE vulns.

3 ambigs.

7. OWASP Dependency Check

12 true CVE vulns.

3 true NON-CVE vulns.

2 ambigs.

17 falses

Unfortunately OWASP Dependency Check is currently unable to handle .NET’s property substitution (e.g., when a *.csproj file references “Directory.Build.props”), a common convention for developers maintaining these files. This causes some frustrating false positives, such as reporting that “Google.Protobuf:$(GoogleProtobufVersion)” is vulnerable to CVE-2015-5237.

OWASP Dependency Check also considers version 0.61.0 of the .NET MySqlConnector package to be vulnerable to 14 CVE’s – these are certainly all false positives. This is probably happening because Dependency Check considers version “0.61.0” to come before releases from MySQL’s popular version 5.x series against which many CVE’s have been filed over the years. However, version “0.61.0” of this package is less than 10 months old, making it impossible that it’s vulnerable to these ancient CVE’s.

Scanning .NET and Nuget Conclusion

Our own offering looks compelling in the .NET space. We were one of the top performers in this mini-benchmark, with Snyk, Whitesource, and the open source OWASP Dependency Check tool also providing reasonable results. We were surprised to see Sonatype and Dependabot perform so poorly here. Software developers currently using the popular open source “Dotnet Retire” tool for this problem should definitely consider other options.

As always with benchmarks such as these and security tools in general your mileage can vary a lot based on the tools you’re using and your own particular context. I think a lot of companies become complacent with their existing tools. Similar to with the smoke detectors in your house, it’s a good idea to benchmark your existing tools periodically, just to ensure that they are still working properly!

Last piece of advice: Have .NET software? Give MergeBase a closer look!

Introducing CodeGreen for Bitbucket

Recommended pre-reading:
Intro to SCA – Software Composition Analysis (

Atlassian Marketplace Link:
MergeBase CodeGreen (


One of the main challenges with known-vulnerabilities is how they mess with standard software lifecycles. A lot of traditional quality engineering relies on the old saying, “if it’s not broken, don’t touch it.” Known-vulnerability announcements for popular open source libraries completely go against that, since they are discovered and announced more or less at random. A good known-vulnerability SCA solution needs to deal with three very different cadences through which known-vulnerabilities will manifest themselves in your software:

If you’re serious about reducing open-source known-vulnerabilities within your software assets, CodeGreen is a tool for getting real results company wide. CodeGreen puts known-vulnerability software composition analysis (SCA) scans directly in front of software engineer eyeballs. A lot of application security work is done by following checklists and invoking security tools and uploading artifacts to cloud URLs during coding and reviewing tasks. CodeGreen short circuits all that by inserting itself directly into your company’s software engineering workflow (as a Bitbucket plugin). From there CodeGreen can inject a range of interventions customized to your corporate application security policy, from low-friction informational reports all the way to outright blocking. These interventions help you quickly get all of your software engineering teams onto the same page.

By attaching directly to the enterprise source-control system (as a Bitbucket plugin) GodeGreen is able to improve application security posture across the board for an entire organization. Your application security will improve within hours after your local Bitbucket administrator installs the CodeGreen plugin through Atlassian’s marketplace.

Vulnerabilities Arrive On Different Cadences

  1. New vulnerability announcements. Your application is not broken, in fact it’s working great! Clients love it. Management is happy. But a known vulnerability has been discovered and published that could be exploited by criminals and bring your brand down. You have to fix it! You must upgrade the insecure library to a safer version.
  2. Accidental vulnerability import (“developer-as-vector”). Under this scenario one of your developers unwittingly introduces a bad library version (that contains known-vulnerabilities) into one of your systems. Just because a “known-vulnerability” is known to the cyber security world at large does not mean it’s known to your own development staff!
  3. That terrifying first scan. This scenario is essentially a combination of the above two scenarios, albeit after several years of unmonitored vulnerability accumulation. The experience of running a first vulnerability scan can be so overwhelming and demoralising for staff that good SCA tools must account for this and provide strategies to manage the first scan.

CodeGreen is a unique tool in the SCA space in that it provides mitigations, reports, and controls designed specifically for these 3 cadences. The rest of this blog post goes into those capabilities in-depth.

For New-Vulnerability-Announcements: Add A Little Friction (Cadence #1)

Developers need to be aware of how newly discovered vulnerabilities affect their systems, but finding time to address these is always a balancing act based on risk, urgency, and other priorities. This is where CodeGreen can apply a little friction.

For Developer-As-Vector: Slam On The Brakes! (Cadence #2)

For cadence #2 (developer-as-vector), once awareness is in place, vulnerabilities should never come into software via this vector. The vast majority of software vulnerabilities are announced alongside a patched (fixed) release of the library. This means developers should never introduce vulnerable libraries into a software project unless such is absolutely unavoidable. This is where CodeGreen can slam on the brakes.

Managing That Terrifying First Scan (Cadence #3)

A lot of security tools are sold and marketed based on a simplified models of their operation – the tool is presented similar to a flashlight. Turn on the light, see into the darkness. But under the hood the tool might offer dials and controls and subtleties to users to help make its operation more successful. CodeGreen is no exception here!

Under ideal operation CodeGreen would be configured to apply maximum friction to encourage developers to eliminate all vulnerabilities, but that’s not tractable for most organizations, at least not at first.

To help make CodeGreen more practical we allow repository administrators to adjust the CVSS thresholds at which the various CodeGreen mitigations become active:

We recommend setting these to more permissive values during your initial rollout, and tightening them to more restrictive values as your teams’ application-security maturity improves.

For example, in the beginning you might want to enable only the CodeGreen double-push friction and set it to a CVSS 9.0 threshold and disable everything else. Make it an overt term goal to clear out all 9.0 vulnerabilities and above.

(But always enable “block-net-new-vulnerabilities” because that’s the dreaded cadence #2!)

Once you’ve achieved that, increase the “double-push” control to use a CVSS threshold of 8.0, so it catches more vulnerabilities.

Meanwhile, enable the “requires dual-approval” control (a much higher friction compared to double-push) and set that one to 9.0.

The end result here is interesting: any newly announced vulnerabilities will suddenly dramatically slow down development teams. The developer has a choice: find someone to approve their work, leaving the vulnerability in place, or just patch the brand new 9.8 vulnerability and avoid the dual-approval.

Which would you choose?

It’s a lot like thoroughly cleaning a house methodically from top to bottom: once a given room is clean, you can lock its door to prevent any additional mess from occurring in the already cleaned room. Similarly here you can clean out all the 9.0’s and above, and then “lock the door” on them by turning on the dual-approval control.


GodeGreen improves application security posture across the board for your entire organization by embedding open-source known-vulnerability scans directly into your centralized git source control. Your application security will improve within hours after your local Bitbucket administrator installs it!

Intro to SCA software composition analysis


Why do we need SCA software? Well, that is a long story:

Commercial and industrial software is now primarily constructed from components. Open source components, to be exact. Open source software licenses dramatically decrease business frictions that arise from incorporating and integrating software developed by external entities. No more contract negotiation or in-house legal review!

Add to this the fact that many software use cases are more or less identical across systems: http connectivity, encryption, spell checking, transaction management, database object mapping, unit testing, etc. The end result is predictable: in less time than it takes to read the 2-clause BSD open source software license, your developers are copying externally developed software libraries into your proprietary systems. Because: why not? The license allows it, and developers achieve their objectives with fewer bugs and time to spare.

Software developers can now easily obtain pre-fabricated high-quality software libraries to help implement significant portions of their software. Your colleagues only need to write a small amount of glue code to wire these libraries into the larger system. Software, like automobiles, is now made mostly from parts.

But unlike cars, the supply chain in the software world is complete mayhem and chaos. Consider this common clause found in the majority of open software licenses:

Unless required by applicable law or agreed to in writing, Licensor provides the Work on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND

Risk management would flag this as a major risk and procurement  policies would not allow such clauses to stand in a legal review. But the advantages of open source software outweigh the risk of running without warranties. You get what you pay for. But there are costs. People often say there is no such thing as a free lunch – the same is true of open source software.

The Upgrade Problem

Some people, when confronted with a problem, think “I know, I’ll use an open source software library.” Now they have two problems.

When I build my systems, I choose specific versions of open source libraries to incorporate into these systems. These versions quickly become stale as open source authors continuously update and evolve their libraries, issuing new releases periodically. Worse, you’ll start to find vulnerabilities in them.

At first glance, the problem looks pretty simple and straight-forward. Somewhere in my build script my software will contain a line like this:


If I’m using JavaScript / NPM, then it would look a little different, but essentially the same:

"dependencies": {

   "apache-struts2": "^2.5.22"


To avoid any vulnerability to the infamous cyber security bug that took down Equifax in 2017, I just have to change the “<version>” line in the text above to this:


Then I click “save” and “build” in my coding editor and voila! My software is now safe. In the NPM example things are even better thanks to that “^” character. The “^” symbol tells the build script to upgrade the library to 2.5.23 automatically.

However, despite how simple the example above appears, in actual practice this problem is a complete f’ing nightmare. For several reasons:

  1. How do I even find out that the libraries I’m using have updates available?
  2. My system currently operates correctly (to the best of my knowledge). Could a library update break my system (regression risk)?
  3. Sometimes libraries change their own calling protocols and requirements in subtle or even not-so-subtle ways. How much work will I need to do updating my glue code to integrate a particular library’s newest version into my software system?
  4. Related to item #3, do the library authors themselves have any recommendations regarding long term plans? For example, the authors of the popular “Apache HttpClient 3.x” library decided they hated maintaining it and rewrote the library completely from scratch. They actively encouraged consumers of their library to switch to their new rewrite (“Apache HttpComponents 4.x”), and stopped all maintenance of the older library, but unfortunately switching to this newer version required significant effort for consumers.
  5. Does the current version of the library I’m using have any critical security flaws in it? Normal bugs prevent or perturb normal usage patterns, but I’ve already established that the library operates correctly within my system, and so I’m not too concerned about normal bug fixes. Security bugs are a whole different animal, since they often allow malicious users to cause the library to misbehave in ways that can degrade or even breach and exploit the larger running system.
  6. Are any of the critical security flaws widely known to the public at large? E.g., are they referenced by specific CVE (Common Vulnerability and Exposure) advisories within the U.S. Government’s NVD (National Vulnerability Database)? Upgrading library versions that are associated to CVE records should be considered a high priority, since cyber security breaches via these vectors are often perceived as engineering negligence by the public.
  7. Can we confirm exploitability based on our current configuration? If we can prove our specific setup is non-exploitable, that can buy us time to postpone the upgrade for now. But sometimes even establishing non-exploitability requires more work than simply upgrading the library.
  8. Bear in mind we must tackle this problem repeatedly for every library currently incorporated into our larger software system. Most minimally useful commercial systems will bring in at least 30 libraries; I figure the average is around 80 libraries; and I’ve personally seen systems that contain more than 300 distinct libraries.
  9. Some practioners recommend upgrading libraries when new library versions contain useful features that you would like to incorporate into your system, especially if such new features would allow you to delete some of your own code. I am on the fence on this matter, since in my opinion the maxim “if it ain’t broke, don’t fix it” outweighs this. However, should a library update happen to obviate code you are using in a different library, allowing you to completely remove one of the library dependencies from your system, I do recommend taking that upgrade. Good luck ever noticing such obviations, however.

The list above enumerates the tensions and problems we face when upgrading software components.

How to manage open source software vulnerabilities and license risk?

So what are people doing about it? First hand “in the field” I’ve seen three different approaches applied to this upgrade problem.

  • PURE MANUAL BEST EFFORTS. Under this approach the engineering team tries their best to keep library versions up to date when possible, and they try to keep an eye on any associated CVE records in the NVD database through google searches and peripheral awareness. END RESULT: typically these systems are severely stale and rife with vulnerabilities.
  • AUTOMATED ALWAYS UPGRADE EVERYTHING ALWAYS. These systems are less affected by CVE’s or other known-vulnerabilities, since known-vulnerability announcements tend to correspond to version updates, and systems under this regime take in updates immediately. This approach does not deal well with incompatible library upgrades, and such usually end up in a “Pure Manual Best Efforts” pile. END RESULT: these systems tend to have fewer known vulnerabilities, but they can be vulnerable to broken builds and regression bugs. They are also vulnerable to supply-chain attacks such as the event-stream NPM attack that occurred in late 2018.
  • TOOL ASSISTED SOFTWARE COMPOSITION ANALYSIS. Engineering teams can use SCA (Software Composition Analysis) tools to tackle the upgrade problem. Despite their name, SCA tools should really be called recall notifiers, since that is their primary function: to determine all public recalls associated with any of the software component versions referenced in a given system. These tools operate similar to the computer at your car dealership when the dealer types in your VIN and determines if your car has any outstanding recalls for any of its constituent parts. SCA tools immediately surface all library versions within your system that correspond to item 6 of my list above, helping software engineers prioritise their upgrading efforts to focus on the most urgent library updates.

SCA tools sometimes include additional features such as copyright license analysis and staleness checks. MergeBase’s own SCA toolchain focuses exclusively on the recall problem.

In summary, the “Upgrade Problem” is a fundamental tension inherent to any software development practice that builds on reuseable software components. The problem is not easy to resolve, but ultimately some libraries MUST be upgraded. Personally I recommend tying the library upgrade decision to two factors: first, consider the library version’s current cyber security risk profile, and second, consider if the library’s own development team is relatively active and responsive.

In a nutshell, leave the library version alone (do not update it) if the following two factors hold (“if it ain’t broke”):

  1. The library is actively maintained.
  2. There are currently no public known-vulnerability security advisories tied to the version my system is using.

Otherwise, upgrade the library! In particular, if factor #1 no longer holds, migrate as soon as possible to an actively maintained competing library. Dead open source libraries like httpclient-3.x and apache-axis are notorious for accumulating CVE’s, and emergency migrations with such defunct libraries become high-effort and high-risk – a terrible combination.

Risk management for your open source software

An SCA tool (such as MergeBase Detect) is critical for determining if a library should be upgraded. In my own experience the “upgrade problem” is simply not tractable for manual best-effort approaches, and always-upgrade is too much work with too little benefit.

There’s one major caveat though. If you’ve been using the “PURE MANUAL BEST EFFORTS” approach for a long time, you need to both don a safety mask and buckle your seat belt before first running an SCA tool against your system. The initial report is going to be intimidating and overwhelming.


Too much fun with “git pull –rebase”

Note:  this article refers to “git pull -r” and “git pull –rebase” interchangeably. They are the same command, except the merge-preserving variation can only be specified via the long form: git pull --rebase=preserve


I’ve long known that “git pull –rebase” reconciles the local branch correctly against upstream amends, rebases, and reorderings. The official “git rebase” documentation attests to this:

‘Note that any commits in HEAD which introduce the same textual changes as a commit in HEAD..<upstream> are omitted (i.e., a patch already accepted upstream with a different commit message or timestamp will be skipped).’

Thanks to the git patch-id command it’s easy to imagine how this mechanism might work. Take two commits, look at their patch-ids, and if they’re the same, drop the local one.

But what about squashes and other force-pushes where git patch-id won’t help? What does “git pull -r” do in those cases? I created a series of synthetic force-pushes to find out. I tried squashes, merge-squashes, dropped commits, merge-base adjustments, and all sorts of other force-push craziness.

I was unable to confuse “git pull –rebase,” no matter how hard I tried. It’s bulletproof, as far as I can tell.

Investigating ‘git pull –rebase’

The context here is not a master branch that’s advancing. The context is a feature branch that two people are working in parallel, where either person might force-push at any time. Something like this:

The starting state. Evangeline and Gabriel are working together on branch ‘feature’. Note: ‘evangeline/feature’ is actually Evangeline’s local ‘feature’ branch, and ‘gabriel/feature’ is Gabriel’s local ‘feature’ branch. Clone it from here, or recreate it using the script below.

git init
echo 'a' > a; git add .; git commit -m 'a'
echo 'b' > b; git add .; git commit -m 'b'
echo 'c' > c; git add .; git commit -m 'c'
git checkout -b feature HEAD~1
echo 'd' > d; git add .; git commit -m 'd'
echo 'e' > e; git add .; git commit -m 'e'
echo 'f' > f; git add .; git commit -m 'f'
git checkout -b gabriel/feature
echo 'gf' > gf; git add .; git commit -m 'gf' --author=''
git checkout -b evangeline/feature HEAD~1
echo 'ef' > ef; git add .; git commit -m 'ef''
git push --mirror [url-to-an-empty-git-repo]

The Experiment

In each scenario Evangeline rewrites the history of origin/feature with a force-push of some kind, usually incorporating her own ‘ef‘ commit into her push. Meanwhile, Gabriel has already made his own ‘gf‘ commit to his local feature branch. For each scenario we want to see if Gabriel can use “git pull –rebase” to correctly reconcile his own work (his ‘gf‘ commit) against Evangeline’s most recent push.


  1. We assume Gabriel has correctly setup remote tracking for his local feature branch. This is a reasonable assumption, since git sets this up by default when a user first types “git checkout feature”.
  2. We only tested Git v2.14.1 and Git v1.7.2 for this experiment. Perhaps “git pull –rebase” behaves differently in other versions.
  3. Important: we only use “git pull –rebase” (or -r).  Some people claim “git fetch; git rebase origin/master” is equivalent to “git pull -r”, but it isn’t.

Force-Push Scenarios

For each scenario we are on Gabriel’s local branch feature. The graph on the left shows both the state of origin/feature (thanks to Evangeline’s force-push), as well as the state of Gabriel’s local feature and how it relates to Evangeline’s force-push.  The graph on the right shows the result of Gabriel typing “git pull -r”.

A scenario is deemed successful if “git pull -r” results in Gabriel’s gf‘ commit sitting on top of origin/feature.  Since Gabriel does not push back in these scenarios, his ‘gf‘ commit remains confined to his local feature branch.

  1. origin/feature rebased (against origin/master)
    This is the canonical example of why we prefer “git pull -r”.  The rebase notices that older commits ‘d‘, ‘e‘, and ‘f‘ on Gabriel’s feature branch are patch-identical to the rebased ones on origin/feature, and thus it only replays the final ‘gf’ commit.

    Result: Success!

  2. origin/feature squash-merged (with origin/master)
    This is the rebase + squash combo meal.  Evangeline takes all work on feature, squashes it down to a single commit, and rebases it on top of origin/master.  She probably did this via “git merge –squash.” I did not expect “git pull -r” to be able to handle this, but I was wrong.

    Result: Success!

  3. origin/feature squashed in-place
    This is the classic squash. Evangeline types “git rebase –interactive origin/master”.  In the interactive screen she marks the first commit as “pick” and every other commit as “squash” or “fixup”.  This squashes feature down to a single commit, but leaves the merge-base alone (commit ‘b‘ in this case). I also did not expect “git pull -r” to handle this one, but I was wrong here, too.

    Result: Success!

  4. origin/feature dropped a commit
    For some reason Evangeline decided she wanted to drop commit ‘e‘ from origin/feature. She ran “git rebase –interactive origin/master” and marked every commit as “pick,” except commit ‘e‘, which she marked with “drop”.  I expected “git pull -r” to erroneously bring commit ‘e‘ back.  I was wrong.  Running “git rebase” instead of “git pull -r” did bring commit ‘e‘ back, and so there is obviously some deeper intelligence inside “git pull -r” enabling the correct behaviour here.

    Result: Success!

  5. origin/feature lost their mind
    I have no idea what Evangeline was trying to do here.  If you look closely, you’ll see she reversed her branch (‘ef’ is now the oldest commit), she squashed the middle two commits, and she adjusted the merge-base so that origin/feature emerges from commit ‘a‘ on the mainline instead of commit ‘b‘.  This is one serious force-push!  I had no idea what to expect here.  I certainly did not expect “git pull -r” to nail it, but it did.

    Result: Success!

  6. origin/feature went back to how things were (undoes the rewrite)
    Evangeline, either through her reflog or her photographic memory, happened to remember that origin/feature previously pointed to commit ‘325a76a.  Here she force-pushes origin/feature back to ‘325a76a‘ to undo her push from scenario 5. The command to do that is useful to know:  “git push –force origin 325a76a:refs/heads/feature”. Staring in awe at how “git push -r” did the right thing for scenario 5, all I could do was continue to stare when it did the same here. (Note: Gabriel’s start-state here is scenario 5, not the original start-state).

    Result: Success!

Conclusion: Time To Revise The Golden Rule

Supposedly, the golden rule of git is to never force-push public branches.

Of course I would never force-push against ‘master’ and ‘release/*’.  As a git admin, that’s always the first config I set for a new repo:  disallow all rewrites for ‘master’ and ‘release/*’.

But all public branches?  I find force-pushing feature branches incredibly useful.

Industry has arrived at a compromise: defer the rewrite to the final merge. Bitbucket, Gitlab, and Github now offer “rebase” and “squash” flavours of PR merge. But it’s a silly compromise, because the golden rule itself is silly.  Instead of building complex merge machinery to dance around the golden rule, I think we’d be better served by reworking the rule itself. Three reasons:

  1. Force-pushes are useful!  Public amends, squashes, and rebases help us make better PR’s for code review.
  2. What is the actual point of the golden rule?  Are we trying to prevent lost work on the mainlines (e.g., ‘master’ and ‘release/*’)?  If that’s the point, then we’re much better off setting appropriate branch permissions on our central git server for those branches.
  3. Is the point to prevent the spaghetti graphs caused by default “git pull” behaviour?  In that case a better golden rule would be never use default “git pull” and always use “git pull –rebase”, since it avoids spaghetti graphs, while allowing history rewrites.

I propose a new golden git rule (in haiku form):

We never force-push master
or release. But always,
for all branches: git pull -r

Alternatively, you can make “git pull -r” the default behaviour:

git config --global pull.rebase true

Git graphs in this article were generated using the Bit-Booster – Rebase Squash Amend plugin for Bitbucket Server.

Discover More from MergeBase

Open Source Protection

Stay on top of the real risk of open source at any time.

Avoid false positives and get sophisticated upgrade guidance based on risk, compatibility and popularity.

More on Continuous Protection

Add RunTime Protection

Detect and defend against known-vulnerabilities at runtime. The only SCA to do so.

The quickest way to respond to an imminent threat like log4j with CVE-2021-44228.

More on Run-time Protection

Shift Left Now

CodeGreen is an early-warning defence for your in-house development and integrates directly into GitHub and BitBucket

More on BitBucket and Github apps