Java is one of the most popular languages in software development—which means hackers are always trying to exploit what vulnerabilities they can in Java libraries. That’s why it’s important to find tools that allow you to scan your code for known vulnerabilities.
Our software composition analysis (SCA) tool is especially proficient in finding known vulnerabilities in your Java code—and for generating a very low amount of false positives.
But we’re not the only Java scanner out there.
In this article:
Java Scanning Tools Compared
We wanted to see how other tools on the market compare in terms of their abilities to catch vulnerabilities in Java code, so we compared a few tools on the market by pointing their free trial solutions at the very popular Spring Framework open-source project. This is a fairly complex project—and one that has several known java vulnerabilities.
NOTE: We wanted to see each tool’s immediate scanning capabilities (without setting up CI/CD pipelines) so we focused on tools that allow us to use command lines to scan our target library.
The high-level results:
| Dependency-Check | SonaType | Snyk | Mend | MergeBase | |
|---|---|---|---|---|---|
| CVEs flagged | 43 | 0 | 26 | 25 | 15 |
| Ease of deployment | Easy | Difficult | Easy | Easy | Easy |
| Command line results | Inferior: must view results page | N/A | Inferior: must use app GUI | Superior: summary of results in command line | Superior: summary of results in command line |
Here’s an in-depth look at what each tool provided us:
Dependency-Check
First up is Dependency-Check (OWASP), a free but excitable tool. Dependency-Check is easy to install and can be used completely locally—and it’s a common starting point for teams who want to level up their application security.
We ran the Spring Framework through Dependency Check, and here’s what it returned:
That’s a pretty busy screenshot, but the main takeaway is that Dependency-Check flagged 226 CVEs, many of which were simple duplicates. When we de-duped the results, we were left with only 45 unique vulnerabilities.
SonaType Nexus
Sonatype’s Nexus was created to help developers manage components, binaries, and build artifacts across their entire software supply chain. Because we didn’t want to set up a pipeline product (SonaType Lift), we uploaded a .war file of the Spring Framework project and uploaded it to the nexus page.
Within a few minutes, we got an email from Sonatype with the following report:
Strangely, the scan turned up zero components, and therefore zero vulnerabilities—this was obviously a bum scan for Sonatype.
SNYK
Snyk is a popular tool in the premium software composition analysis space: they support a host of integrations, they maintain a thorough documentation bank, and their command line tool is very easy to install.
However, the process of getting CVE information from Snyk for this experiment was less than intuitive. We attempted to scan for CVE’s in the Spring Framework project using three different approaches (namely, a command to output a JSON file, a command to make the output available in our account’s user interface, and their IntelliJ IDE plugin).
While each method provided component names along with developer guidance for remediating vulnerabilities, it wasn’t easy to get the actual CVE names. This was important, as we wanted to cross-reference Snyk’s findings with the findings from other tools and more easily identify false positives.
The JSON output did include CVE names, however, the file was 153,552 lines long and listed 3305 CVEs—but only 26 of them were unique. And although the UI dashboard displays each component name, you need to click into each individual component to display CVE info.
Fine granularity is helpful, but a general snapshot of scan results would have been helpful, too. This is a lot to wade through just to get a comprehensive understanding of the scan we just performed.
Mend (formerly WhiteSource)
But we had no such qualms with Mend. We could easily install the program and conduct a scan from the command line. Furthermore, Mend generated a concise summary table within the command line once the scan was complete, giving us a quick outline of their findings followed by a long list with further information on each vulnerability—a helpful presentation of both high-level and granular information!
Mend returned every unique vulnerability that Snyk with one exception (CVE-2022-40155)—more on that later.
MergeBase
Finally, we scanned the Spring Framework project using MergeBase, resulting in a concise overview right in the command line:
And of course, the MergeBase GUI gives a more detailed report:
MergeBase finds the fewest number of vulnerabilities out of all of the tools—let’s see why.
Which Java scanner did the best job?
Now let’s pull all this together to see which tool did the job best. We’ll look at how the tools performed in terms of their ability to catch vulnerabilities, their ease of deployment, and their ability to generate useful, easy-to-understand reports.
Catching Vulnerabilities
Dependency-Check reports far more vulnerabilities than any other tool we checked—many of which are false positives, including CVE-2010-0538, which affects MacOs Apple Java. Unfortunately, this is in keeping with Dependency-Check’s reputation for generating false positives. Developers often start with Dependency-Check as a Java scanner, but given false positives’ negative effects on both security operations and team morale, it’s understandable why growing teams usually move on to a premium solution.
The disagreements between Snyk, Mend, and Mergebase are mostly rooted in where each tool locates the vulnerability. For example, MergeBase reports the undertow component as being vulnerable, whereas Mend and Snyk do not. But both Mend and Snyk report xnio as being vulnerable, whereas MergeBase does not. This is because xnio is actually a dependency of undertow, so what’s going on is a difference in where in the dependency tree the vulnerability is being caught.
So why does MergeBase report only 15 vulnerabilities, while the other premium tools report 25 and 26? This is because Mergebase doesn’t report dev dependencies such as Jython (which “provides implementations of Python in Java”)—instead it flags dependencies that actually end up in your production build.
Ease of deployment
SonaType was the only one we had any trouble getting working, and it was more intrusive than the other tools in terms of being able to scan quickly and privately.
Dependency-Check worked just fine, but only after using the IntelliJ Gradle plugin. Deploying MergeBase, Mend, and Snyk was a quick and easy process all around.
Quality of reports
The ideal Java scanner will provide an immediate, high-level summary of the scan right in the CLI as well as thorough and well-organized reports in the GUI. While none of these tools generated reports that were difficult to understand, some reports were more immediately intuitive than others.
Dependency-Check was the clear loser in this respect: the results only really make sense if you see the report in the HTML page the tool generates when it completes a scan—their command line output is a wall of word salad.
Snyk was better but still leaves much to be desired in terms of intuitive reports. For example, Snyk found 72 medium violations—but does this represent 72 different CVEs or 72 different components are vulnerable? (We did some research: it’s the latter.) Furthermore, Snyk has the issue we mentioned previously: a massive granular report with no intuitive summary.
Mend does a bit better: they provide a summary in the CLI once it finishes a scan. But when needing JSON output, separates out the component information into one file and the vulnerability information in another. This means that in order to parse any real meaningful results, you need to stitch the two together by their matching SHAs. That’s a lot of extra work to understand the scan you just performed.
In some ways, MergeBase has an unfair advantage here: while Snyk and Mend are broader cybersecurity solutions, MergeBase’s software composition analysis tool was purpose-built to overcome these kinds of inconveniences developers experience when using other solutions. To that end, we generate a concise summary in the command line when a scan completes, as well as a thorough report in our GUI.
Run accurate Java code analysis with MergeBase
Selecting the right Java code analysis tool is critical to protecting your product, your organization, and your users. We built MergeBase to help you rapidly find true vulnerabilities (not false positives), remediate them, and keep your applications secure.
If you’d like to see how MergeBase can keep your Java applications secure,start a free trial today and experience its advanced capabilities firsthand.
