We recently completed version 1 of our .NET scanner. The goal is to take any .NET project and determine if it contains any references to software libraries with known-vulnerabilities.

For this blog post we thought we’d take our scanner out for a spin and see how it compares against the competition.

Results Summary:

MergeBase – 18 vulnerabilities, 0 false positives.

Snyk – 7 vulns and 5 false, or 4 vulns and 0 false (depends on scanner setup).

WhiteSource – 12 vulns, 0 false.

OWASP Dependency Check – 12 vulns, 17 false.

Dotnet Retire – 2 vulns, 0 false.

Sonatype – 0 vulns, 0 false.

Dependabot – 0 vulns, 0 false.

 

Methodology:

We chose the .NET Orleans project as a subject project to be scanned. It’s active, complex, and it builds successfully (August 6th, 2020, master = 2e10856f7b7ed9443c). We also liked how this project contained a mix of Nuget styles (e.g., older “packages.config” style as well as the newer “<PackageReference/>” style).

We type “dotnet build” before scanning. This way scanners can use the generated “obj/project.assets.json” files to supplement their scan data if they want to, and “dotnet build” is such a critical step for building any .NET project that we think it’s safe for an SCA tool to assume this command has completed successfully.

As for comparing results, we count CVE’s. If the scan outputs 1 or 300 or 9,000,000 hits against CVE-2018-8292, we count that as a single CVE. We then do a quick “desk check” to categorize the result as either a true-hit, a false-negative, or an ambiguous result (where it’s hard to say one way or the other). The “desk check” is very much based on my own decades of experience as a software engineer – I encourage others to rerun these scans and see if they agree or disagree.

Because this is a .NET scan, we ignore any results the scanners find from other file-types lying on the file system (e.g., “VotingWeb/wwwroot/lib/jquery/jquery.min.js”). We do, however, count results found from nuget references into other language artifacts (e.g., “GPSTracker.Web/packages.config” contains a nuget reference to “<package id=”bootstrap” version=”3.0.0″ targetFramework=”net45″ />” in its packages.config file – we’ll count this.)

Here is the exact sequence of steps:

1. git clone https://github.com/dotnet/orleans.git

2. dotnet build

3. Deploy The Scanners!

4. Validate the results.

A note about ambiguous results:

We classify some results as ambiguous. This means there’s definitely some smoke, so we can’t rule immediately it out as a false negative after examining the metadata, but on the other hand, there’s enough uncertainty to also make us uncomfortable considering it a true hit.

Example:

The vulnerability references “bootstrap” in the scan report but the CVE description talks about “bootstrap-sass”. Maybe? Or in another case the CVE description starts out with the words (in all caps) “DISPUTED”.

Results:

I’ll save the best for first! Here’s what MergeBase finds:

1. MergeBase

18 vulnerabilities found (and two ambiguous hits).

Drop the scanner into the Orleans subdirectory. Type “java -jar mergebase.jar .” and the results are pretty straightforward: 2 critical CVE’s, 5 high ones, and 11 mediums. A quick spot-check of the metadata looked good (no false positives and two ambiguous results).

2. Github Dependabot

Zero vulnerabilities found.

Dependabot not doing too much here, despite being a Microsoft product (albeit recently acquired):

3. Dotnet Retire

Two vulnerabilities found: CVE-2018-8292 and CVE-2018-8416. MergeBase also found these two among the 18 vulnerabilities it identified.

4. Sonatype

Zero .NET vulnerabilities found!

Sonatype does detect a small handful of JavaScript vulnerabilities (since Orleans contains things like “VotingWeb/wwwroot/lib/jquery/jquery.min.js”), but nothing for .NET. To be fair, their scanner instructions did say “you must copy all .NET packages you depend on into the zip file you are scanning beforehand.” I typed “dotnet build” and zipped the result (660MB). As far as I’m concerned, I was doing them a favour by even zipping up orleans post-build in the first place – no other scanner required that.

Note-to-self: Probably MergeBase should also scan those JavaScript packages! (Our current logic looks for NPM and Yarn lock files, but maybe it’s time to roll up our sleeves and consider scanning raw *.js and *.min.js files, too.)

5. Snyk

It’s Complicated!” The problem with Snyk is that there’s two different ways to invoke the Snyk scanner, and each way returns wildly different results.

Snyk Approach #1 – Github Integration:

15 vulnerabilities found. 5 of those are false positives (all because Microsoft.NETCore.App was flagged as a dependency, but it’s not). 3 are ambiguous. 7 are true hits.

A few NPM and Docker vulnerabilities also found, but seeing as this bakeoff is only about .NET we ignored those.

Snyk Approach #2 – Command Line Invocation:

7 vulnerabilities found. 3 are ambiguous, leaving 4 true hits, including 1 true hit that Snyk approach #1 above did not find (CVE-2020-1469).

No NPM or Docker vulnerabilities found via this approach.

6. Whitesource Bolt

12 true CVE vulns.

3 ambigs.

7. OWASP Dependency Check

12 true CVE vulns.

3 true NON-CVE vulns.

2 ambigs.

17 falses

Unfortunately OWASP Dependency Check is currently unable to handle .NET’s property substitution (e.g., when a *.csproj file references “Directory.Build.props”), a common convention for developers maintaining these files. This causes some frustrating false positives, such as reporting that “Google.Protobuf:$(GoogleProtobufVersion)” is vulnerable to CVE-2015-5237.

OWASP Dependency Check also considers version 0.61.0 of the .NET MySqlConnector package to be vulnerable to 14 CVE’s – these are certainly all false positives. This is probably happening because Dependency Check considers version “0.61.0” to come before releases from MySQL’s popular version 5.x series against which many CVE’s have been filed over the years. However, version “0.61.0” of this package is less than 10 months old, making it impossible that it’s vulnerable to these ancient CVE’s.

Conclusion

<tbd>

Introducing  CodeGreen for Bitbucket 


Recommended pre-reading:
  Intro to SCA – Software Composition Analysis (mergebase.com)

Atlassian Marketplace Link:
  MergeBase CodeGreen (marketplace.atlassian.com)


Introduction

If you’re serious about reducing open-source known-vulnerabilities within your software assets, CodeGreen is a tool for getting real results company wide. CodeGreen puts known-vulnerability software composition analysis (SCA) scans directly in front of software engineer eyeballs. A lot of application security work is done by following checklists and invoking security tools and uploading artifacts to cloud URLs during coding and reviewing tasks. CodeGreen short circuits all that by inserting itself directly into your company’s software engineering workflow (as a Bitbucket plugin). From there CodeGreen can inject a range of interventions customized to your corporate application security policy, from low-friction informational reports all the way to outright blocking. These interventions help you quickly get all of your software engineering teams onto the same page.

By attaching directly to the enterprise source-control system (as a Bitbucket plugin) GodeGreen is able to improve application security posture across the board for an entire organization. Your application security will improve within hours after your local Bitbucket administrator installs the CodeGreen plugin through Atlassian’s marketplace.

 

Vulnerabilities Arrive On Different Cadences

One of the main challenges with known-vulnerabilities is how they mess with standard software lifecycles. A lot of traditional quality engineering relies on the old saying, “if it’s not broken, don’t touch it.” Known-vulnerability announcements for popular open source libraries completely go against that, since they are discovered and announced more or less at random. A good known-vulnerability SCA solution needs to deal with three very different cadences through which known-vulnerabilities will manifest themselves in your software:

  1. New vulnerability announcements. Your application is not broken, in fact it’s working great! Clients love it. Management is happy. But a known vulnerability has been discovered and published that could be exploited by criminals and bring your brand down. You have to fix it! You must upgrade the insecure library to a safer version.
  2. Accidental vulnerability import (“developer-as-vector”). Under this scenario one of your developers unwittingly introduces a bad library version (that contains known-vulnerabilities) into one of your systems. Just because a “known-vulnerability” is known to the cyber security world at large does not mean it’s known to your own development staff!
  3. That terrifying first scan. This scenario is essentially a combination of the above two scenarios, albeit after several years of unmonitored vulnerability accumulation. The experience of running a first vulnerability scan can be so overwhelming and demoralising for staff that good SCA tools must account for this and provide strategies to manage the first scan.

CodeGreen is a unique tool in the SCA space in that it provides mitigations, reports, and controls designed specifically for these 3 cadences. The rest of this blog post goes into those capabilities in-depth.

For New-Vulnerability-Announcements: Add A Little Friction (Cadence #1)

Developers need to be aware of how newly discovered vulnerabilities affect their systems, but finding time to address these is always a balancing act based on risk, urgency, and other priorities. This is where CodeGreen can apply a little friction.

For Developer-As-Vector: Slam On The Brakes! (Cadence #2)

For cadence #2 (developer-as-vector), once awareness is in place, vulnerabilities should never come into software via this vector. The vast majority of software vulnerabilities are announced alongside a patched (fixed) release of the library. This means developers should never introduce vulnerable libraries into a software project unless such is absolutely unavoidable. This is where CodeGreen can slam on the brakes.

Managing That Terrifying First Scan (Cadence #3)

A lot of security tools are sold and marketed based on a simplified models of their operation – the tool is presented similar to a flashlight. Turn on the light, see into the darkness. But under the hood the tool might offer dials and controls and subtleties to users to help make its operation more successful. CodeGreen is no exception here!

Under ideal operation CodeGreen would be configured to apply maximum friction to encourage developers to eliminate all vulnerabilities, but that’s not tractable for most organizations, at least not at first.

To help make CodeGreen more practical we allow repository administrators to adjust the CVSS thresholds at which the various CodeGreen mitigations become active:

We recommend setting these to more permissive values during your initial rollout, and tightening them to more restrictive values as your teams’ application-security maturity improves.

For example, in the beginning you might want to enable only the CodeGreen double-push friction and set it to a CVSS 9.0 threshold and disable everything else. Make it an overt term goal to clear out all 9.0 vulnerabilities and above.

(But always enable “block-net-new-vulnerabilities” because that’s the dreaded cadence #2!)

Once you’ve achieved that, increase the “double-push” control to use a CVSS threshold of 8.0, so it catches more vulnerabilities.

Meanwhile, enable the “requires dual-approval” control (a much higher friction compared to double-push) and set that one to 9.0.

The end result here is interesting: any newly announced vulnerabilities will suddenly dramatically slow down development teams. The developer has a choice: find someone to approve their work, leaving the vulnerability in place, or just patch the brand new 9.8 vulnerability and avoid the dual-approval.

Which would you choose?

It’s a lot like thoroughly cleaning a house methodically from top to bottom: once a given room is clean, you can lock its door to prevent any additional mess from occurring in the already cleaned room. Similarly here you can clean out all the 9.0’s and above, and then “lock the door” on them by turning on the dual-approval control.

Conclusion

GodeGreen improves application security posture across the board for your entire organization by embedding open-source known-vulnerability scans directly into your centralized git source control. Your application security will improve within hours after your local Bitbucket administrator installs it!

VANCOUVER, BC – Will cybercrime cause $1 trillion in damage to an already-vulnerable economy by 2021? Not if MergeBase Software Inc. has anything to do with it. The company has just announced it raised $500,000 funding for its best-in-class cybersecurity product — helping it ramp up sales and distribution. The funding round officially closed on March 19.

“I’m impressed that during this unprecedented crisis, business leaders and investors are able to ‘keep calm and carry on’, continuing to invest in leading-edge technology to solve critical problems like cybercrime,” says MergeBase CEO Oscar van der Meer. 

The current COVID-19 crisis and social distancing measures will only accelerate the move to a fully digital economy. In this new environment, cybersecurity for digital assets and IT will be even more mission-critical to business and governments.

“So many technology-powered companies are built on open-source code and third-party apps — which is a quicker, easier and cheaper way of building software,” he explains. “But those savings come with a cost, exposing organisations using these applications to data breaches.

“Integration with external apps already causes up to 24 percent of all cybersecurity breaches — and that’s only going to grow,” he says. “MergeBase is a best-in-class solution to boost the immune system of enterprises around the world.” MergeBase’s app-security solution detects more vulnerabilities than any other tool on the market.

Enterprises are already boosting purchases of app security solutions, from 5 percent in 2019 to 60 percent by 2024, according to a Gartner report. “It’s why we’re expecting to see big growth in our business.” 

MergeBase’s solutions are aimed at large enterprises. Their customers include a government agency that processes trillions of dollars of payments every year. 

The company’s co-founders bring a combined 50 years of experience in the financial industry, and a wealth of knowledge about identifying and dealing with vulnerabilities in technology; for instance, van der Meer was a senior executive at Central1, the central financial facility and trade association for the B.C. and Ontario credit union systems. 

Investors in the current funding round include Lisa Shields and Western Universities Technology Innovation Fund (WUTIF) and Maninder Dhaliwal.