MergeBase loves java. In fact, you could say that java is our mother tongue, so we strive to create the best java scanner in the business. For this blog post, I thought I’d validate our confidence in our java scanning and see how it compares to the competition. Join us in this process & see for yourself.
The number of unique CVE’s flagged by each tool:
- MergeBase: 15
- Snyk: 26
- Mend: 25
- OWASP: 43
- Sonatype: 0
The scan project I chose is the massive Spring open source project, which you can find here: https://github.com/spring-projects/spring-framework. It’s a wildly popular framework with ~50,000 GitHub stars, has a very active community with many pushes to production per day, and is a Gradle project with multiple sub-modules. This project should give all of the SCA tools a good run for their money with a reasonable amount of complexity.
Let’s start off by pulling the project into our IDE with the command:
I chose the following commit in order to ensure we get a nice batch of juicy vulnerabilities: 001b2636d645acac12d78d0e548ad71ef708af26 ( October 17, 2022 ), scanned on November 8.
git checkout 001b2636d645acac12d78d0e548ad71ef708af26
I then built the project with:
Choosing the Java Scanning Tools to comparate
Now let’s gather our SCA tools. I decided only to use tools with free trials because while I am cheerful, I am also cheap. I also want to be able to try the tools without having to book a demo, as I like discovering things on my own with no pressure. And lastly, I do not want to have to set up any ci/cd pipelines. It’s dark, rainy, and too November-ish to be going down that rabbit hole today, so I will only test tools with easy to understand instructions & with low overhead in terms of infrastructure, which basically means: command line tools only, please.
I begin with SonaType Nexus. In order to not have to set up a pipeline product (SonaType Lift), I uploaded a .war file of “my” project (spring) as requested. I generate this file with the following command:
jar -cvf test_spring.war *
Then upload it to the nexus page from my freshly made account.
Within a few minutes, I get an email back stating that my report is ready:
No components were found; thus, no vulnerabilities were found, which is obviously not ideal. So, what’s up with that? Did I not create my .war appropriately? I play around with which folder to compile my .war from and start realizing that the “email your file” system is intrusive & a bit embarrassing if you need to keep trying different ways of making it work. I like to hide my general confusion in deep, dark anonymity personally. I found no way of getting any results and finally hit my free trial limit, so I called it a day with them.
Next up is Dependency-Check (OWASP), a free but excitable tool that tends to lean towards type 1 errors and thus is often way too eager to lift up the alarm with false positives, which can be rather time consuming to take care of from a dev standpoint. What I do like about DC, though, is that it’s easy to install and can be done completely locally without having the sense that “they” are watching you.
/opt/dependency-check/bin/dependency-check.sh --enableExperimental --project test-spring -s .
Once again, I have no vulnerabilities. This one surprises me, as it is running the opposite of what I expected: no results rather than the usual onslaught of false positives… this can’t be right. I look online to find out more about Gradle submodule Dependency-Check scans and found out that OWASP has an IntelliJ Gradle plugin. 10 minutes later, once my plugin is installed & IntelliJ restarted, I run the following command:
I finally got a fruitful result! 226 CVE’s found, which is much more like what I expected from DC. There are quite a few duplicates, so once those are removed, we are left with 45 unique vulnerabilities.
Let’s move on to Snyk. I’m generally fond of them as they have loads of integrations, a great documentation bank, and an easy to install command line tool (npm install snyk). I have to be honest, though, and say I was a bit frustrated with them during this experiment. I tried to get cve info in 3 ways from them:
- By using their cli to output json:
snyk test --skip-unresolved --json-file-output=snyk_output.json
- By using their cli with the command to have it available in my account UI
- By using their IntelliJ IDE plugin
Each method did bring up component names & methods in which to remediate the situation, but darn, was it hard to get the actual cve names. These cve names were important to me in order to be able to cross reference with the other tool findings as well as to have something to look up in order to verify that it wasn’t a false positive.
The only place I was able to find actual cve names was in the json that was output in the first command (snyk_output.json) a json file that was 153,552 lines long & had 3305 cve’s listed, but only 26 unique! In the UI dashboard, I need to click through each component name in order to get the cve info instead of just having a summarized table like WS & MB do.
I appreciate the fine granularity but would have liked to see my results more generally, too.
Mend (formerly WhiteSource) is also quite easy to run from the command line with:
I have to say that I quite like their output at the end of the scan: a table with a quick outline of their findings followed by a long list with further information on each vulnerability. This way, you can see from both the bird’s & the mole’s eye view.
Leaving the most familiar for last, MergeBase, I run it with the following command:
java -jar /Users/vw/Sites/MergeBase/mergebase-staging.jar --name=spring_blog . --all
Here we get a nice, compact, concise summation of the scan with a more detailed report found in the GUI.
MergeBase finds the fewest number of vulnerabilities out of all of the tools, so it’s now time for some analysis to see why.
MergeBase: 16 violations, 5 warnings
Snyk: 14 high, 72 medium, 32 low with snyk monitor
4 medium,1 low with IntelliJ plugin
25 vulnerable paths with snyk test
Mend: 19 High, 10 medium, 2 Low
OWASP: Vulnerable Dependencies: 13, Vulnerabilities: 18
Sonatype : 0 vulns.
In the table below, you will find the 11 vulnerabilities that all of the tools agreed on.
Snyk and Mend agreed on all vulnerabilities but 1 (CVE-2022-40155). The disagreements between Snyk/Mend and Mergebase are regarding dev dependencies and the actual component that was vulnerable. For example, MB reports the undertow component as being vulnerable, whereas Mend & Snyk do not, but Mend & Snyk report xnio as being vulnerable, whereas MB does not. What’s interesting is that xnio is a dependency of undertow, so what’s actually going on is a difference in where in the dependency tree the vulnerability is being caught.
MergeBase finds the least amount of vulnerabilities of the bunch, but that is because Mergebase does not report dev dependencies such as Jython (which “provides implementations of Python in Java”) but rather just flags dependencies that end up in your production build.
Dependency-Check, meanwhile, comfortably finds the most vulnerabilities & in fact, flags 21 more cve’s than either Snyk, Mend, or Mergebase. One of these flags is CVE-2010-0538, which affects MacOs Apple Java. These are exactly the kind of false positives that we at Mergebase strive to avoid in our scans.
- For ease of deployment, I would say that SonaType was really the only one I was unable to get working, and it was overall quite intrusive in terms of being able to scan quickly & privately. Dependency-Check worked well, but only after I got the IntelliJ Gradle plugin. The rest of the tools were easy to deploy.
- In terms of ease of understanding the results, I would have to say that none of them were really very difficult, but many did have their own ways of reporting the results in terms of how they grouped the information. That is when you see that Snyk found 72 medium violations, that makes you wonder: does this represent having 72 different cves flagged or does it mean that 72 different components are vulnerable (after some research, it is the latter). The json output of the tools is often way too verbose and honestly seems quite repetitive. Mend, when needing json output, separates out the component information into 1 file and the vulnerability information in another file, so in order to parse any real meaningful results, you need to stitch the two together by their matching SHAs. Snyk has a nice GUI, but personally, I would really like to see a nice summary table at the end of the scan like Mergebase & Mend.
- Looking at the command line output at the end of the scan, I really like Mergebase & Mend the best. Sure, GUI’s are great, but sometimes you just want a quick and dirty answer at the end of the scan in your terminal console. Dependency-Check results only really make sense if you see the report in the html page that they emit at the end of a scan (found in the projects build/reports folder), as their command line output is just a wall of word salad. Snyk, as stated before, really bummed me out on this end. The only really comprehensive way of seeing your results is in their GUI with snyk monitor, but even there, you do not get a quick recap of the findings, which is quite frustrating.
I hope that you now have a better idea of what’s out there for java scanning and the pluses and minuses of each.