Software supply chain attacks have escalated over the years, with a 650% increase in 2021. The problem with third-party libraries is that they are complicated, hard to review, and constantly move, with millions of libraries and hundreds of versions of each. These vulnerabilities account for 80 to 90% of an application.
The response time for fixing these security issues can take anywhere from a month to two years. Rolling out patches can take even more time, depending on the complexity of the systems and whether you’re developing your software or buying someone else’s software and waiting for a patch. This vulnerability window can be deadly, as an attacker can weaponize these attacks and deploy them in weeks, while it may take more weeks or months for even the best organizations to roll these patches out.
This is where MergeBase’s Dynamic Application Surveillance and Hardening (DASH) comes in. Java Runtime Protection allows you to block a Java library or a function within it so that it can’t execute. It allows you can monitor your applications, look at which functions in third-party components are being used, and lock down the ones that aren’t being used or block that library or that individual method in case there’s a CVE on it.
Java Runtime Protection allows you to provide protection inside of an application without going through the development life cycle to be able to do something. It also allows you to deploy hot patches to software that you didn’t write, such as commercial off-the-shelf software.
Learn more about SCA Runtime Protection
In this webinar, Jim Manico, Julius Musseau, and Shannon James Smith will present:
- Most known exploits (CVEs) are patchable, but some aren’t
- Patching can take time, easily weeks or even months
- Java Runtime Protection buys time and offers a solution without patching
If you want to learn more about SCA Runtime Protection and how it can help you protect your software supply chain, check our whitepaper about it and book a demo with us today.
Video Transcript
Shannon James Smith: Good morning everybody. Thanks for joining us today at the first of six SCA Runtime Protection sessions. We’re doing these in quick 20-minute sessions so that nobody gets bored or consumes too much of their time on any one day, but I think you’ll find the topics and discussion fairly interesting since nobody in the world offers Java Runtime Protection except MergeBase.
And I learned about this a few years ago and introduced it to some of my friends in the security industry, and they said, “Wow, this is amazing.” And I was thinking, “Wow, this is so amazing. It should be patented.” And it is patented now. So we’re coming out with it, and we’re bringing it to market, and it’s patented, granted in patents in the US and Canada. And we’re going to find out what it is, why it matters. And we’ve got a great guest with us today, Jim Manico. And I say, hi, Jim.
Jim Manico: Hi everybody. Hello. How are you doing today?
Shannon James Smith: Great, great. And Jim does application security training for a number of prestigious clients. He’s been very active in OWASP over the years, and we’re incredibly happy to have him today to drill our CTO and co-founder about what this runtime stuff is and why it matters. And say hi, Julius.
Julius Musseau: Oh, hey. Hi everyone. Yeah, so happy to be here and to see everyone here.
Shannon James Smith: So we also have Julius’s co-founder, Oscar, on the line, our CEO. Say Hi, Oscar.
Oscar van der Meer: Hi.
Shannon James Smith: So I’m going to go through just a few slides just to put this in perspective in case anybody’s not in tune with what we’re really talking about here. But the software supply chain has become an issue in the recent past five years that people are actually paying attention to due to some high-profile hacks. But more recently, it’s really exploded and started to go exponential, and this has always been an issue.
This is not something that just came about in the last 5 or 10 years. It’s always been an issue. Hackers have been using supply chain stuff forever. However, recently, about five or six years ago, it started to become more of an issue, and we started to see exponential growth in these attacks, as you can see from the chart on the right.
And then more recently, last year or two years ago now, we saw over a 600% increase in these attacks. And then Gartner predicted last year that this rate would not continue that fast, but another a hundred percent a year, basically increasing in these attacks. And CIOs and CISOs are now becoming very aware of this, and they realize they can’t skirt the risk and responsibility in this area, and the vast majority of them know that they’re vulnerable to these types of attacks.
We’ve seen a massive impact on some of these attacks. The Equifax attack a couple of years ago that hit Apache Struts caused a 5 billion loss of market cap within a couple of days. And that loss of market cap did not bounce back. Its competitors grew over the next few years. Equifax’s market cap did not grow back. And basically, what we’re talking about here is the majority of software, it’s the majority of software in a modern application is third-party components.
So people write their own custom code; maybe that’s 10 to 20% of any business application. And then the rest, 80/90% of that application, is externally third-party sourced code, generally open source. And that reduces cost and time to market for people and doesn’t have everyone building their own wheels for their cars. They go to wheel manufacturers. And this is a massive complex ecosystem basically, and it’s really grown exponentially along with the attacks.
The public repositories in GitHub went from less than 50,000 in 2009 to over 28 million by 2020, 3 years ago. So we got an increase of 61000% in the last decade of public repositories, which has resulted in millions of software libraries being used, hundreds of versions of each of those, multiple distribution paths, and then all kinds of ways to get into the supply chain and cause trouble basically. And the response time for fixing these security issues can literally go from a month to two years. So it can widely vary.
And then, to compound this, JavaScript is now the number one language in the world. However, Java is still the number one enterprise software language in the world. And Java, as it turns out, has more critical vulnerabilities than all other leading languages combined. And this is part of the reason that we created SCA Runtime Protection, and the first implementation of Runtime Protection is protecting Java.
You can see that basically, when a new software library comes out or a new CVE comes out with a new security vulnerability, developers want to get the new stuff and upgrade it, of course. And if they have a very fast, agile production line, they can sometimes do that in a pretty quick amount of time.
But rolling that out and getting that into production can take a lot more time depending on the complexity of the systems, and whether you’re developing your own software or you’re just buying someone else’s software and waiting for a patch, this vulnerability window can be deadly. I mean, an attacker can weaponize these attacks and deploy them in weeks, and it may take more weeks or months for even the best organizations to roll these patches out.
And so Java Runtime Protection allows you to actually block a library or a function within a library so that it can’t execute. And it turns out that while we use a lot of libraries, we use very few functions in those libraries. So the majority of functions in these libraries that are being used in people’s software aren’t used. They’ve basically unlocked doors in a very big house, and no one’s taken the time to look at them and say, “We should lock all these doors we don’t use.”
Most applications just run with open doors. And so now we have this ability for you basically to monitor your applications, look at which functions in third-party components are being used, and the ones that aren’t being used, you can either proactively lock them down because they don’t need to be used. So lease privilege, you lock them down, or there’s a CVE on them, and you don’t want to take the risk until the patch is available so you can block that library or that individual method.
So really, I’m going to stop here, and I’m just going to hand this over to Jim. Jim knows about Runtime Protection, but he’s also been learning about it. I have in the last couple of months here and learning more about it, and this is our first session to really drill Julius and Oscar about what this is, why it matters, and how it can help people.
Jim Manico: Absolutely. I think one of the biggest problems I see is a lot of the folks who are building their own custom software; there are a lot of different technologies that address updating third-party components at build time, at development time. We know that already, there’s a huge industry around that, but to provide protection inside of an application that lets me provide protection without going through the dev life cycle to be able to do something.
Or even better, to be able to deploy hot patches to software that I didn’t write, like commercial off-the-shelf software similar that I don’t have a lot of development control over. There’s not a lot we can do about that. I mean, there are other approaches to Runtime Protection in the web application and API space.
Some of that does address the supply chain problem, but there are not a lot of folks who address it completely like this, where the main focus of MergeBase is Runtime Protection specific for the supply chain in a way that lets me apply this to commercial off-the-shelf software.
Those are a lot of claims that not a lot of, not a lot of competitors even begin to approach. And this is compelling because the problems that we see in third-party libraries are accelerating, and the dependence on them is really bad right now. So to be able to handle these particular third-party library component security problems across custom and commercial software for different security teams, that’s a really big deal, Shannon and Julius.
Julius Musseau: It’s really thinking heading into the webinar here today, thinking about Log4j and Equifax. Now, I think we’ve gotten to a point where if people say Log4j in a cybersecurity context, brains just turn off. We’ve heard the word Log4j maybe a little too much in the last year, but if you really drill down to the technical aspects of those attacks, both the Struts attack, the Equifax attack, and the Log4j attack, what’s really interesting there in the Equifax case, it was an error handler, a bad content type string had been provided.
So when you have an HCTP request, it’s always going to provide a content type. It’s going to say, “Oh, I would like a PDF.” Or “I would like an HTML page or a CSS page,” These are the content types. So what happened here was there was an error handler that was waiting to see, “Oh, that’s a bad content type. I don’t know how to handle that. Here, I’ll pass it off to this error handler.”
And so a very technical aspect of the attack was that the attacker noticed, “Oh, that error handler is going to pass off its error handling routine to this very custom specialized templating library.” Somehow the attackers just noticed this amazing series of coincidences that they were able to leverage there.
So both the Log4j bug and the Equifax bug they’re miracles; they’re incredible to see. And then their consequences are so severe in the Log4j case what the attacker was taking advantage of was this JNDI class loading ability at Runtime, string substitution in the log messages again.
So I mean, what I’m saying here, is they sound very deep and technical and esoteric, and they are, but what’s interesting is these very esoteric, very special little pieces of code are leveraged by attackers to completely take over the system.
And I think SCA Runtime Protection becomes very important here because we are able to see, “Hey, this is a really weird code path that you would not normally be using, and a bunch of these weird exotic code paths are all being strung together to create these cataclysmic exploits.”
Jim Manico: Right on.
Julius Musseau: I know that with Log4j, we did actually see some hot patching using Java; what’s it called? The Java profiling agent, you know, where you could say of WAFS?
Jim Manico: The profiling agent or the successor to the profiling agent. But it’s basically the instrumentation of the JVM that you’re hooking into. I’ve also seen web application firewalls approach this problem with some hot patching, with some success, but limited success. So what I think is special is the full application wrapper, and I’m not even explaining it well. Would you get a little more technical, Julius?
Would you mind digging into, again, I think, the other competitors in the Java space? Especially, there’s web application firewall technology where I can hot patch certain classes of attacks in a pretty, and not a super mature way, in a pretty standard, rule-based, regex-ish kind of way at the WAF level.
And my opinion, I’m not huge; I mean, I use them, and I put salt on my food sometimes, but I don’t like it. I know it’s not good for me, but I do it, whatever. Anyways. And then there’s the Java instrumentation, a similar API, and I don’t like that at all because it’s really an invasive way to mess with the runtime application. It’s extraordinarily invasive. A lot of problems can happen, so I’m hesitant about this space. I’m hesitant when anyone says, “We do Runtime Protection, including you, Julius.”
So I want to hear what you’re doing technically at a deep level, so I can recommend this to all of my customers. Convince me to recommend your Runtime Protection, that you’re not going to tweak my instrumentation an API, or you’re not going to hand me a bunch of regex to solve the problem. I need something better, Julius, dammit. All right. All right. Sorry, I’m sorry.
Julius Musseau: I’m like, good to know.
Jim Manico: Sorry, sorry, sorry, sorry.
Julius Musseau: Ah, yeah. Right. So I mean, our approach actually is pretty simple in that we take the Java libraries, which are files, and we go through them function by function or method by method for people in the Java world. And we insert a little hook, a little sort of callback that will keep track of what functions are being called by which other functions. But it’s very simple. We’re not looking at the data going into the functions.
We’re not looking to be like, “Oh, is a bad string coming in or a bad array of bites?” We’re just literally looking, “This function’s being called.” Because we noticed in the Log4j case, for example, oh, this JNDI message substitution method being invoked. That’s all you need to know. You don’t need to know the data coming in. Just the fact it’s being invoked at all is very suspicious.
Jim Manico: Absolutely.
Julius Musseau: But what we do is we prepare the JAR falls; we essentially rewrite them to inject this tiny bit of bite code to keep track of this stuff at… What would you say? On disk we rewrite them on disk. When the system, so you bring your system down, you shut it down, you apply the MergeBase, which then rewrites all the JAR files, and then you bring your system back up, and away you go.
And then now it’s reporting its usage back to the MergeBase dashboard. Whereas when the Log4j thing came out, you would see people attaching an agent and doing this hot patching, but the moment the system came down, you’d have to do it again. It wouldn’t persist between restarts and problems like that.
Jim Manico: Can I ask about the rewrite that you’re doing? What bite code are you injecting into a JAR to give you visibility to provide these kinds of defenses?
Julius Musseau: So the bite code specifically is, it’s detailed in a lot of detail in our patent, but at a high level, what we do is we stuff in about, I think it’s about 1500 lines of code into the JAR file. And then what happens is that every time a public method or protected method is invoked, the very first thing it’s going to do is go call this little bookkeeping method.
this thousand lines of code that we’ve injected to say, “Hey, just so you know, I was invoked.” And then this bookkeeping method, this bookkeeping class sort, keeps a tally of what’s going on and then reports back to the MergeBase dashboard on a periodic basis, like once every 10 minutes or so.
Shannon James Smith: So they don’t need the source code. This can be an application that they bought off the shelf, and they don’t have to have source code access to get MergeBase Runtime Protection running.
Julius Musseau: Yeah, we’ve applied it to Jira, for example, and we don’t have the source code to Jira, just our own running internal Jira system. So we just applied it to it. It is fun to watch all the vulnerable parts of Jira being executed or not because we’re able to see that.
Jim Manico: Let me ask you a quick question, Julius. When you’re instrumenting, when you’re providing bookkeeping on individual method calls, is there a big performance hit? Because I know you’re not doing a full generalized Runtime Protection. This is specific to supply chain problems. So have you benchmarked or looked at the performance or any other impact?
Julius Musseau: Yeah, in our own benchmarking, it’s less than 2% with the default application. I mean, this drives Oscar crazy. I’m always like, “But caveats if you go in depth.” So we do have the ability to turn on a deeper inspection, a deeper monitoring per class. So if you apply that, then that will slow that class down by about 10%. But out of the gates, we’re talking around 1 2% degradation in our own benchmark.
Shannon James Smith: So basically negligible for any kind of production load. Okay. So the attendees know if you guys have questions, you can join the talk, or you can send them through the Q&A. Go ahead.
Jim Manico: So for a 1% not noticeable hit, you’re able to now instrument every individual method call. Look it up in a very lightweight lean bookkeeper, essentially. So at the individual method level, you could make security decisions, and this is without hitting the profiling or instrumentation part of the JVM. This is without rules. This is just you’re basically injecting a very lightweight code into the JAR so you can do this individual instrumentation. Is that a fair explanation?
Julius Musseau: Yeah, absolutely. Yeah. We don’t use agents at all. So the logic of the JAR files has already been changed ahead of time before the system starts running.
Jim Manico: This is fantastic when I get asked about this because a lot of people are looking into Runtime Protection, like the dev cycle is just, even with DevOps, too slow. People want to do something immediately, and they lean on WAF a lot, but it’s limited in how to do this. So this is a very unique and innovative way to provide this method-level inspection. This is something I have not heard anyone doing at this level that I know of.
Shannon James Smith: Yeah, it’s definitely a new feature that really applies to almost anyone because anybody running any business application has Java, and they’re at risk. And what happens is you get a new CVE that pops up, and it may take your team two weeks to apply the patch or to update the library, or it may take them two months, or it may take them to a year, or they may never be able to do it.
And the point is, with Runtime Protection, you can go in there and monitor this stuff and/or block it so that you’re not at risk of someone exploiting one of these new CVEs against you. I want to just open it up. Do we have any questions you want to just talk about? You can feel free to unmute at this point.
Jim Manico: Try answering this one, Julius. It’s from Ahmed, but I think I got the answer. Can I try?
Julius Musseau: Yes, please.
Jim Manico: So Ahmed, like MergeBase, has Runtime Protection. That’s what we’re talking about in this webinar that’s unique and lets you patch live commercial software. The other part of MergeBase is a software composition analysis scanning engine, which of course, deals with JavaScript, which supports JavaScript.
So like a normal code scanning engine, you would scan your development code base, and it would pick up JavaScript and alert you to any third-party library vulnerabilities in the JavaScript that you’re using. That traditional software composition analysis scanner.
But again, what we’re talking about on the webinar, in addition to that, scanning technology MergeBase has Runtime Protection for applications, including wrappers around commercial software to block individual methods and more like during Runtime when a problem’s discovered. That’s my answer; how’d I do?
Shannon James Smith: So just on your question Ahmed, we’re not doing Runtime Protection for JavaScript yet because it is covered by our patent, but we haven’t developed that feature. And the main reason is, as the slide I showed before, that Java has more critical vulnerabilities than JavaScript and all the other languages that everyone uses combined.
So it’s definitely the area of the most need where that vulnerability window can really kill you. But we certainly look forward to developing Runtime Protection and rolling it out for JavaScript and C and C++ and C# in the future. And the patent does cover that.
Jim Manico: And Shannon, again, MergeBase today has a solution to address supply chain problems with JavaScript applications. That’s fantastic. And this is at the dev scanning level, so this is area support, an area of capability that MergeBase has had from the beginning of the company. I want to make sure that we get that across.
Shannon James Smith: Yeah, this Runtime Protection thing is basically the end of the whole MergeBase production line. We start with the dev, work with the dev sec, and then, but the unique thing about MergeBase is it actually offers an ops solution for security. That gives you that visibility from the beginning of the design and development phase all the way through the development pipeline, which is what you’re talking about.
And we support 12 languages doing that. And then we give you visibility actually into ops for many languages, but we give you this protection capability right now just for Java. Is that a fair way to summarize it, Oscar?
Jim Manico: I’ll take it.
Shannon James Smith: Where we wanted to make these sessions digestible and not take your whole day, and it’s kind of a big topic because we kind of went around a lot on a lot of different things that we’re going to go into more detail in the future sessions. But yeah, we’re going to have one session that talks about WAFs and RASP and how that compares to Runtime.
The next session, let me share my screen here, is about hardening your applications. So the way most people will think about using Runtime Protection is, “Oh no, there’s a new CVE out. Now I’m going to reactively block that method in the Log4j library so that I’m not vulnerable to that CVE, and I’m not rushing to patch now.”
But you can also use Runtime Protection proactively and monitor your applications and see which functions in which libraries are actually being used by the application and the other, whatever, 92% of functions that aren’t being used after a safe period of six days or six weeks or however long you want to monitor that app, you can now shut those down and lock those doors and harden your applications from future zero days or unknown vulnerabilities that haven’t been publicly announced yet.
And this is the ability to dramatically reduce your application security attack surface, and nobody offers this type of function. As I said, WAFs and RASPs are trying to do it in some fuzzy ways, but this is a very precise, exact way. This should not give you any false positives ever.
Julius Musseau: To add to what Shannon is saying here, to make it maybe a little more concrete. For example, in our Jira instance where we applied for the MergeBase SCA Runtime Protection, and we were doing our scrum meetings, and we are doing all our Jira tickets and marking them in progress and backlog and done. So we were exercising Jira like a normal team would. It was our Jira.
What we noticed is I think it was around 20% of the JAR files were never used, like never invoked, never loaded. But they’re just sitting there; they’re just sitting there on the classpath, waiting to be loaded one day.
Shannon James Smith: Well, in the essence of time, we started this thing about a minute late. So we’re going to end it about a minute late, but we’re going to try to keep this on time, respect everyone’s schedule, and we hope you guys will come back and join us in two weeks to dive into the ins and outs of using SCA Runtime Protection to harden your applications and reduce your kind of unknown attack surface in your app space. So thank you, Jim and everybody, for joining us.
Jim Manico: Appreciate it.
Shannon James Smith: Thank all the attendees. Thank you.
