Explore the intersection of security and productivity in software development. Discover effective tool sequencing, understand the importance of developer engagement, and learn strategies to minimize security-related work overload. In this insightful video, MergeBase CEO Oscar van de Meer, CTO Kelly West, and key engineer Delan Elliott delve into the nuances of application security and tool sequencing. These insights are crucial for any software development team aiming to enhance productivity without compromising security.
Balancing Security and Productivity
A strategic approach to application security, focusing on the sequencing of security tool implementation and prioritizing high-value tasks, is essential. By utilizing advanced technologies like Dynamic Application Surveillance and Hardening, organizations can effectively balance security and productivity in software engineering.
Are you looking to enhance your software development process with efficient and effective security measures?
How accurate scanning tools save you time and effort - Video Transcription
Oscar: Hi, everyone. My name is Oscar van de Meer. I’m the CEO of MergeBase. We have here Kelly West, who’s the CTO, and Delan Elliott, who’s one of our key engineers.
We want to talk about something related to what’s impacting productivity for engineers. And so security obviously also impacts, is one of those elements that potentially negatively impact productivity for engineers. And obviously, we want to minimize that.
What are some of the security implications of productivity for engineers, Kelly?
Kelly: The nature of application security is to look for things that need to be changed or addressed and then address them. So, the very nature of application security is going to generate some work. But there are different phrases that are used. Certainly, an industry standard would be talking about vulnerability fatigue, or I’ve heard developers themselves refer to it as technical toil, which tells you how they feel about it.
But that, application security can generate to-do lists that are long and not necessarily prioritized effectively. Sometimes, the to-do list is to actually try and diagnose why something showed up in application security tests and doesn’t even indicate how you can remediate it. Application security generates work to do.
And the level of value isn’t always clearly indicated. And I think that that’s a key aspect of why you get the fatigue or the sense of toil.
Oscar: So, do you feel that there’s kind of a mismatch between the amount of work that is being requested to be done and the level of time that is made available to do it? Is that kind of a key issue?
Kelly: Yeah, I think it’s one of the key elements. I mean, there’s always a function of more work than time. I think that’s just par for the course. But I think that is a clear indicator of the value of the work, like everybody wants to deliver value when they’re doing work. And so, based on the priority of the work and the value it delivers, developers will perceive some of this work as having value.
Oh, I’ve addressed this vulnerability. We were previously unaware of this, but clearly, it’s worth addressing. And some of it feels like busy work. It doesn’t feel like it’s high value. And telling the difference is the key.
Oscar: So, work is being requested, but it’s not clear to whoever is doing the work how valuable that work is for their organization. And I mean, I killed this one vulnerability. And so basically reduced the risk for the organization. It doesn’t feel like an exciting kind of achievement. Is that part of it?
Kelly: Well, I think there are a couple of aspects to it, and you just identified one. One is it’s really important from leadership all the way down to talk about why addressing vulnerabilities is adding value.
So, in other words, that it is security work doesn’t mean it’s low-value work and just comes out of an automated to-do list. And so it’s really important to talk about it in the right terms and ensure that people understand the value of it, that’s a broader education sense. But the other part of it is to make sure it really is only valuable.
And you’re getting rid of all the things that could be wasting developers’ time.
Oscar: Yeah, Delan, perhaps you can talk a little bit about where all these application security work items come from?
Delan: First, obviously, technical toil isn’t necessarily a term to be used for all security work. Obviously, we understand that. But when you’re introducing a security program, sometimes there’s a tendency to create too many tasks for yourself in such that you’re using many tools. You want to cover all of your bases. You want to have a comprehensive program.
Let’s say, you know, from the top, it sounds great. Like we’ve, we’ve covered our static application testing. We’ve covered dynamic application testing, and we’ve open-source components. And once you have all of these tools, each of them might generate different levels of tasks or different types of tasks as well as, you know, for instance, you know.
I worked somewhere where we had a testing tool, and when it was first run on the project that, uh, was being tested, it generated 33,000 to-do items in Jira. So. As you could tell, you’re not going to be getting through your 33,000-item to-do list anytime soon.
So, there’s an element of triage. There’s an element of, you know, choosing the type of work that you actually find valuable. And that is sort of the start. And, you know, even before that, we need to choose the right sets of tools.
There are three main types of tools: static application testing, dynamic application testing, and software composition analysis. And you want to introduce these tools at, you know, different levels of the build process. They all test different things.
With static application testing and software composition analysis, you want to insert that right at the beginning of the build process because that might actually reduce the number of problems you find later on in the process, where you would do something like dynamic application security testing. So, you would want to implement tools that can be used earlier in the pipeline to prevent additional issues.
If you’re, if you have implemented all these tools at different parts of the process, you’re going to be, you know, maybe duplicating items that, uh, that might be covered by a different set of tools that you haven’t implemented, that you’ve, that you haven’t finished your to-do lists earlier in the process.
Oscar: And so you’re talking about these 33,000 vulnerabilities. So isn’t that also just not a kind of an implementation issue where the organization didn’t really understand how this tool would behave on their kind of application set? And so should they have not have kind of just started with a prepsor space small thing and then once they’ve built out the understanding.
Delan: In this case, it probably came down to a static application security testing tool; there’s sort of a set of best practices that are recommended by either the person who makes the tool or by industry people who have implemented these tools.
And so if you’re in charge of the security program, you see this set of best practices, and you’re like, this would be great. We should definitely do this on all of our products. The program is new, and you haven’t been following that. You’re soon going to find these best practices have not been followed in all of this legacy code that we have or even some of our new code, you know, some of these best practices don’t actually match up to things that affect productivity or our particular, the way we handle other parts of our security process, these things aren’t necessary.
So if you just implement from best practices, you’re going to find that these to-do items or these, you know, uh, items that need to be fixed are going to exceed far your ability to fix them. So it is about at the highest level of the security team, the CISO or your operation security team that’s implementing the tool has some communication with the developers who are working on the product actively to know what are the real potential problems we’re facing and what are the things that we need to fix immediately. Starting with a subset of things is something that you can do to reduce that right at the start of when you’re implementing one of these tools.
Oscar: previously, you talked about sequencing and which tool to use first because there seems to be logical sequencing.
Can you summarize that again? What is the logical sequence of tools to use from your perspective?
Delan: There are the three tools I mentioned, and the first tool that you would want to have in your sequence of tools would be software composition analysis. Because software composition analysis deals with all the open-source software that you don’t even control.
So before you even started writing any code, all of this open-source software was out there. So, at the beginning of the process, you have to choose which open-source software you’re going to be importing into your software. And so at the start, and that’s going to be, you know, implemented in the build pipeline or even before that, when you’re choosing these packages to go into your product, it’s best to know before you even start if you’re adding a vulnerability or not. So, if you have a source code, then you have essentially eliminated the vulnerability before it even started. So that’s the first step.
The next step would be static code analysis. These are things the simplest thing is no password, no plain text passwords in your code, and then all the way up to looking for SQL injection abilities and, you know, the common types of vulnerabilities. And so those things are going to be found as well, you know, in the build process.
So in your CI/CD pipelines, and then lastly, you know, after, you know, you’ve implemented those tools, and you’ve done the preventative work, then you’ll have dynamic application security testing, you know, it’s going to look for, you know, edge cases or things that have been missed by the other tools, cause you can’t search for them with static analysis, or simply oversights in your own code where you’ve introduced your own vulnerabilities. Which tends to be the least targeted type of vulnerability an organization can have, right?
Because the types of common ones that can be found by static analysis and by open source components are the ones that are actually going to be the most risky to your organization because attackers are going to be targeting those types of vulnerabilities anyway, so you get a two-for-one here where you’re reducing the number of to-do items, and you’re also eliminating the most risk for your organization. So, you know, that sort of sequencing is very valuable.
Kelly: I’ve worked in a young company that was in a really exciting growth stage. They were growing really fast, but they still were not mature in all of their tools and processes, and they put in a software composition analysis tool only. So they did it first. Their intent, of course, would be to implement a full suite over time. But when they had to make the choice of what tool was gonna bring them the most value? They put in software composition analysis first, and it aligns with the stats we see where there was some research I saw that said as much as 70% of the code in an application might come from open-source components, et cetera.
Oscar: A lot of organizations just configure the tool and situate it; it will only look at new pieces of code being written. So effectively, it looks at the code quality as it is being produced right now and not at historic coding issues because that would completely overwhelm it.
In terms of SCA, which is kind of what is the first step that you guys are recommending, are there then smart ways to kind of, you know, even reduce further?
Kelly: A detail that you can’t ignore is paying attention to the capabilities of the tool. So, in particular, for SAS and SCA tools, false positives are a killer. You need a tool with an adequate level of accuracy, not only in identifying true positives, but in reducing the false positives so that they don’t they’re not catching all the vulnerabilities because they catch everything under the sun, whether it’s legitimate or not, but rather really tightly focused on making sure vulnerabilities are actual vulnerabilities, reducing the false positives is key.
In addition to that, some tools will generate a to-do list, but the to-do list is essentially you figuring out your next step on each item in the list, whereas a newer generation tool will provide more meaningful guidance and help a developer understand what their options are, what can they do next. It can be a very high value to have the right guidance so that you don’t have to figure out how to solve a problem. In fact, the solution is there in front of you when the problem is reported.
Oscar: I mean, I’ve heard from some tools; they provide some guidance, but they just provide a list of upgrades to these versions. And I heard somebody in organizations say, well, what we do is we’ll just apply that and then see where it breaks. Are there better approaches, Kelly?
Kelly: Yeah, definitely. An effective tool is going to tell you what versions are available, their level of compatibility with your own software, and their popularity in the marketplace. So, these are all considerations when you’re trying to choose what version to upgrade to. I mean, ultimately, you have to apply it and ensure that it works. You’re never going to get 100% certainty prior to applying it, but you can get good quality information that will help guide you through the decision.
Oscar: To put in a little bit of a plug for MergeBase. We have runtime capabilities that are unique in the marketplace. And so they can actually think also we use a workload for, for engineers further. I’m not sure who if you want to talk about that.
Delan: We have what we call Dynamic Application Surveillance and Hardening. And what that will do is it’ll actually actively monitor your software and see which components are being used, and it will also allow you to block unused parts of the code that are associated with vulnerabilities.
So perhaps you use a vulnerable version of a library, but this library has many functions. It could be something like comments collections, Apache comments collections, something that has many functions. You use three of them, and there’s a vulnerability in a fourth thing that you don’t actually use. So, what MergeBase allows you to do is to actually just block that vulnerable part of the library while still remaining on the vulnerable version, thereby reducing risk and reducing the developer’s having to test a new version of the tool to maintain compatibility. So it gives you another option that can reduce the amount of busy work that you’re doing.
Kelly: I actually think there are huge benefits potentially from this. This is really what makes an SCA tool a next-generation SCA tool: having that runtime visibility into what’s being used and that runtime availability of the ability to block it.
Simples because, to go back to this idea of the problem of false positives, because there are false positives because there’s poor quality information available, and then there are false positives because, while it might be a legitimate vulnerability, it doesn’t apply in the context that the application is because of how the application is using the library.
So, in that case, how do you know that? Well, It’s challenging. It might be tribal knowledge amongst the developers. You might need to get to just the right person who knows that detail. And so that’s not really practical or scalable. Whereas when you look at the Dynamic Application Surveillance and Hardening and where we’re monitoring the application in real time, and you can see what’s actually being used, you don’t have to guess. You can see what the implications are. And even being able to choose to block it there means you can close a vulnerability immediately and then update the library, you know, as that fits with your priorities, etc.
Oscar: So, that Dynamic Application Surveillance and Hardening capability.
What a question I always get is, well, yeah, but how does it work?
Delan: We do very small code rewrites that essentially call the MergeBase tool every time a method is invoked. You know, this does add, you know, a slight overhead. So, you know, we found it adds about a 1% performance overhead.
However, when you compare this to other agent-based tools or things that hook into things like the JVM directly, we actually have much less performance overhead where those tools might have an eight to 10% average overhead, we have a 1% overhead. So that is something to note and that it’s not going to be something that you’re going to see a huge performance cost or your AWS bills are going to go up or anything like that. Small code rewrite. So you will see that. And that will call back to the merge base dashboard with that information. So you only need outgoing connections, no incoming connections from our tools. And so it can be used in a secure context as well.
Oscar: And so, how difficult is it to deploy? Do I have to change my production environment in my deployment process?
Delan: You know, you can provide additional environment variables if you want to, you know, tell different environments apart, and that would just be up to you, but there’s actually no, no changes required to the, uh, to the environment in order to get.
Oscar: That sounds fantastic. So, can I kind of just summarize a little bit? What I’ve heard is that security has an impact on software engineers, and the smart way to kind of reduce that impact is to look at the sequencing of tools. Like you start with SCA, perhaps SAST, and then DAST, then be very careful how you actually implement those tools because otherwise, you also get 33,000 vulnerabilities. Really, take small steps.
It’s what I’m hearing, and then use kind of modern tools that have advanced capabilities around developer guidance and runtime behavior. Did I kind of miss anything from what you guys were talking about?
Kelly: No, I think that that covers it. The sequence is very important, and ultimately, the tools you choose are very important in getting all the way to that runtime awareness and protection. I mean, that ultimately can reduce the vulnerability, fatigue, or technical toil the most overall.
Delan: And then also getting your developers involved in your security process early on because the developers are the ones that are going to have to implement all of the things that you are setting as policies. And so, you know, anything that isn’t strictly defined in a legal context or compliance to a standard, you know, any, you know, wiggle room, you have to change the priorities and, you know, ensure that high-value tasks are getting pushed to the top and you’re not getting a flood of lower value tasks. That is something you want to focus on as well with the implementation of any of the tools.
Oscar: Delan and Kelly, fantastic insight. Thank you very much for sharing that. Thank you very much.
