We recently completed version 1 of our .NET scanner. The goal is to take any .NET project and determine if it contains any references to software libraries with known-vulnerabilities.
For this blog post we thought we’d take our scanner out for a spin and see how it compares against the competition.
- MergeBase – 18 vulnerabilities, 0 false positives.
- Snyk – 7 vulns and 5 false, or 4 vulns and 0 false (depends on scanner setup).
- WhiteSource – 12 vulns, 0 false.
- OWASP Dependency Check – 12 vulns, 17 false.
- Dotnet Retire – 2 vulns, 0 false.
- Sonatype – 0 vulns, 0 false.
- Dependabot – 0 vulns, 0 false.
We chose the .NET Orleans project as a subject project to be scanned. It’s active, complex, and it builds successfully (August 6th, 2020, master = 2e10856f7b7ed9443c). We also liked how this project contained a mix of Nuget styles (e.g., older “packages.config” style as well as the newer “<PackageReference/>” style).
We type “dotnet build” before scanning. This way scanners can use the generated “obj/project.assets.json” files to supplement their scan data if they want to, and “dotnet build” is such a critical step for building any .NET project that we think it’s safe for an SCA tool to assume this command has completed successfully.
As for comparing results, we count CVE’s. If the scan outputs 1 or 300 or 9,000,000 hits against CVE-2018-8292, we count that as a single CVE. We then do a quick “desk check” to categorize the result as either a true-hit, a false-negative, or an ambiguous result (where it’s hard to say one way or the other). The “desk check” is very much based on my own decades of experience as a software engineer – I encourage others to rerun these scans and see if they agree or disagree.
Because this is a .NET scan, we ignore any results the scanners find from other file-types lying on the file system (e.g., “VotingWeb/wwwroot/lib/jquery/jquery.min.js”). We do, however, count results found from nuget references into other language artifacts (e.g., “GPSTracker.Web/packages.config” contains a nuget reference to “<package id=”bootstrap” version=”3.0.0″ targetFramework=”net45″ />” in its packages.config file – we’ll count this.)
Here is the exact sequence of steps:
- git clone https://github.com/dotnet/orleans.git
- dotnet build
- Deploy The Scanners!
- Validate the results.
A note about ambiguous results:
We classify some results as ambiguous. This means there’s definitely some smoke, so we can’t rule immediately it out as a false negative after examining the metadata, but on the other hand, there’s enough uncertainty to also make us uncomfortable considering it a true hit.
The vulnerability references “bootstrap” in the scan report but the CVE description talks about “bootstrap-sass”. Maybe? Or in another case the CVE description starts out with the words (in all caps) “DISPUTED”.
I’ll save the best for first! Here’s what MergeBase finds:
18 vulnerabilities found (and two ambiguous hits).
Drop the scanner into the Orleans subdirectory. Type “java -jar mergebase.jar .” and the results are pretty straightforward: 2 critical CVE’s, 5 high ones, and 11 mediums. A quick spot-check of the metadata looked good (no false positives and two ambiguous results).
2. Github Dependabot
Zero vulnerabilities found.
Dependabot not doing too much here, despite being a Microsoft product (albeit recently acquired):
3. Dotnet Retire
Two vulnerabilities found: CVE-2018-8292 and CVE-2018-8416. MergeBase also found these two among the 18 vulnerabilities it identified.
Zero .NET vulnerabilities found!
“It’s Complicated!” The problem with Snyk is that there’s two different ways to invoke the Snyk scanner, and each way returns wildly different results.
Snyk Approach #1 – Github Integration:
15 vulnerabilities found. 5 of those are false positives (all because Microsoft.NETCore.App was flagged as a dependency, but it’s not). 3 are ambiguous. 7 are true hits.
A few NPM and Docker vulnerabilities also found, but seeing as this bakeoff is only about .NET we ignored those.
Snyk Approach #2 – Command Line Invocation:
7 vulnerabilities found. 3 are ambiguous, leaving 4 true hits, including 1 true hit that Snyk approach #1 above did not find (CVE-2020-1469).
No NPM or Docker vulnerabilities found via this approach.
6. Whitesource Bolt
12 true CVE vulns.
7. OWASP Dependency Check
12 true CVE vulns.
3 true NON-CVE vulns.
Unfortunately OWASP Dependency Check is currently unable to handle .NET’s property substitution (e.g., when a *.csproj file references “Directory.Build.props”), a common convention for developers maintaining these files. This causes some frustrating false positives, such as reporting that “Google.Protobuf:$(GoogleProtobufVersion)” is vulnerable to CVE-2015-5237.
OWASP Dependency Check also considers version 0.61.0 of the .NET MySqlConnector package to be vulnerable to 14 CVE’s – these are certainly all false positives. This is probably happening because Dependency Check considers version “0.61.0” to come before releases from MySQL’s popular version 5.x series against which many CVE’s have been filed over the years. However, version “0.61.0” of this package is less than 10 months old, making it impossible that it’s vulnerable to these ancient CVE’s.
Our own offering looks compelling in the .NET space. We were one of the top performers in this mini-benchmark, with Snyk, Whitesource, and the open source OWASP Dependency Check tool also providing reasonable results. We were surprised to see Sonatype and Dependabot perform so poorly here. Software developers currently using the popular open source “Dotnet Retire” tool for this problem should definitely consider other options.
As always with benchmarks such as these and security tools in general your mileage can vary a lot based on the tools you’re using and your own particular context. I think a lot of companies become complacent with their existing tools. Similar to with the smoke detectors in your house, it’s a good idea to benchmark your existing tools periodically, just to ensure that they are still working properly!