Blog

How to Deploy an Effective Application Security Toolset?

SCA+SAST+DAST+IAST+RASP+.. = JOY ?

spoiler alert: probably not

Application Security Toolset

In this webinar, application security experts Jim Manico (OWASP top 10’s contributor), Farshad Abasi (OWASP Chapter Lead), and Julius Musseau will go over best practices for rolling out an effective Application Security Toolset to your software development and security teams.

How can you best get started, and how can you best optimize your AppSec toolset over time? Watch this webinar NOW!

How can you best get started, and how can you best optimize your AppSec toolset over time?

Are you just thinking of implementing AppSec tools, or are you looking at optimizing your existing toolset? Watch Jim, Farshad and Julius in this webinar to hear:

  1. What combination of tools gives you maximum protection?
  2. What tool gives you the highest value out of the box?
  3. How are threats likely to evolve, and how to use your AppSec tool set to stay one step ahead of cyber advisories?

Webinar’s Highlights

Definition / Acronyms

  1. SAST –  Static Application Security Testing

It looks at the source code. The scanner will investigate the source code of the system, looking for bad coding practices. For example, static application security testing can spot SQL injection or hard-coded tokens and hard-coded passwords.

2. DAST – Dynamic Application Security Testing

It looks at the running applications. If you have the running application, perhaps production, a pilot or a staging environment, the DAST tools enable you to detect security vulnerabilities in running applications using penetration tests. However, it tends to be a very slow process.

3. SCA – Software Composition Analysis (Third-Party Library Scanner)

It is specifically about open source libraries. It looks at all the open source libraries in a software system. For example, protect your business against the Equifax attack or the log4j problem back in December.In other words, It looks at bad software components and lets everyone know you need to upgrade you need not. The SCA tools are essential because open source reusable libraries are here to stay; everyone’s using them. MergeBase has a research work about open source uses. The results show, for example, that Jira has around 80% at this point.

4. IAST – Interactive Application Security Testing

It is about installing a runtime agent in the programming language, which is usually like the profiling or monitoring API of a core runtime engine and instrumentation API, and you will install an agent at the level of instrumentation to see how objects and classes and the like are executed by the runtime engine. So you are installing an agent at the runtime level of the software and watching how the program runs, not from an external point of view, but from an agent that you have installed within the application itself. Get more visibility supposedly into how the app is running.

5. RASP – Runtime Application Self Protection

It provides personalized protection to applications. It uses the app’s data and state to enable it to identify threats at runtime. It automatically isolates and identifies vulnerabilities.

Matrix – Categorize Cybersecurity tools and Information Security Tools

How to Deploy an Effective Application Security Toolset?

What are common/typical challenges teams face when implementing these tools?

The challenges faced are:

  • It lights up a lot of false positive number
  • It requires an expert operator.

Normally, they get flooded with uh findings right like thousands of findings; what do you do, and how do you prioritize them? And the solution is to bring in an expert to tune.

Application Security Best Practices

Three approaches with a different focus:

  • SCA – > External Libraries/ Supply Chain

Analyze applications for know vulnerabilities, and secure your software supply chain.

  • DAST – Test Application Surface

Try to break into production applications. It will catch configuration issues and some of the above vulnerabilities.

  • SAST – Source code, developed in-house

Analyze source code for bad coding practices

These tools complement each other, and the best practice is to us all there.

Ready to start securing your applications?

Preventing Git Rebase Fights

What’s a Git Rebase Fight?

Have you ever experienced this situation?

1. You go to merge your PR (pull-request), but the PR says it must be rebased before you can merge.

merge your PR (pull-request)

2. You rebase it, which kicks off a new build. But the build must complete before you’re allowed to merge.

cannot-merge-build-in-progress

3. The build completes! Yay! But while you were waiting, someone else managed to merge their PR, updating origin/master with their work. Uh oh!

Preventing Git Rebase Fights

4. You go back to step 1 and hope for better luck this time…

Bumping into this loop once or twice a month is not a big deal (especially if step 2 takes under 10 seconds). But sometimes, the situation can become pathological.

Solution: Optimistic Build Status Propagation

You can use $(git diff TARGET...SOURCE | git patch-id) to prevent these rebase fights. This is handy when you know the build is very likely to succeed (e.g., squashes, amends, clean rebases, clean sync-merges).

The technique is called “Optimistic Build Status Propagation” because it uses the output of “git patch-id” as a heuristic to propagate build status to newer commit-ids without requiring the actual full build to finish.  It works like this:

  1. You push a new branch to your central git server.
  2. The push triggers a build on your CI (continuous integration) server.
  3. The build eventually succeeds. The branch’s tip commit is marked with a SUCCESS flag.
  4. You decide to rebase your branch.
  5. During the rebase, your git server notices two things: 1. The commit before the rebase had a successful build filed against it, and  2. The rebase was clean (no conflicts). 
  6. The rebase triggers a build on your CI server.  The CI server sends your git server an IN-PROGRESS notification.
  7. Your git server receives the IN-PROGRESS notification. Because the git server also knows the rebase was clean and also knows the pre-rebase commit had a SUCCESS flag, your git server optimistically marks the new tip commit with SUCCESS instead of IN-PROGRESS.

In other words, the “SUCCESS” flag from before the rebase is propagated to the commit created after the rebase. That’s why it’s called “Optimistic Build Status Propagation”. It lets you merge immediately after the rebase since there’s no need to wait for the build to complete.

This optimistic window is temporary. Only the IN-PROGRESS flag is intercepted and replaced with a SUCCESS flag. Eventually, the CI server will complete the rebased build and send a final SUCCESS or FAILURE notification. These are dutifully recorded, of course, replacing any previous flags filed against the commit.

This technique is invaluable for shops running a fast-forward merge policy alongside a build-all-branches policy. If your builds are even just a little bit slow (e.g., 3 minutes or worse), your staff are probably waging an infinite rebase war against each other. Or they’ve found a better job somewhere else. Or they’ve disabled those merge policies.

If you’re on Bitbucket Server, there is at least one plugin for this: PR-Booster for Bitbucket Server.

pull request Enhancements
Screenshot from PR-Booster‘s config screen running inside a Bitbucket Server instance.

I’m not aware of any pre-baked solutions for this problem on Gitlab or Github; I’ll update this blog post if I become aware of anything.

Optimistic build status propagation in action
Optimistic build status propagation in action: the JOB-809 build above was sent as “IN-PROGRESS,” but PR-Booster intercepted and filed it as SUCCESS instead.

What Causes Git Rebase Fights?

Fast-forward merge policy causes rebase fights.

A fast-forward merge policy only lets PRs merge if they are ahead of origin/master. In other words, PRs must be rebased before they can merge. The policy keeps git history neat, clean and linear by eliminating merge commits. But the policy can also cause rebase fights. I can think of two situations in particular where this happens:

1. High contention for the merge right-of-way and repo is so large that rebases are slow.

I suspect this situation is rare and confined to very large teams.  Rebases are only slow with very large repos, and you’d need at least 50+ engineers targeting the same origin/master before the contention would get high enough.  Monorepos, in particular, may be vulnerable to this.

2. Slow secondary commit validation processes (e.g., builds must succeed before merge, but builds are 3+ minutes).

I suspect this situation is much more common. If your rebase fights are happening because of scenario 1 (very large repo + very large team), then you should probably forget about running a fast-forward policy.  Sorry, but you need those merge commits. In exchange for a messier commit graph, you get improved productivity.  It’s a good tradeoff!

If your rebase fights are happening because of scenario 2 (slow secondary processes), then “Optimistic Build Status Propagation” is available as solid mitigation. Under scenario 2, you can have both a clean commit graph and a productive team!

Triple-Dot-Diff and “git patch-id”

I refer to “git diff A…B” as triple-dot-diff.  When people complain about Git’s usability, the triple-dot operator is certainly one of Git’s blemishes. The operator’s behaviour is inconsistent across various commands (e.g., “git log A…B” does something quite different).

The manual for “git diff” explains the triple-dot-diff like so:

     “git diff A…B” is equivalent to “git diff $(git-merge-base A B) B

Visually, it looks like this:

git diff master branch

This is because the merge-base of master and branch is committed cc603d1, the last commit they had in common before they diverged.  And so “git diff masterbranch” is equivalent to “git diff cc603d1   ee7b565”.

It turns out clean rebases, squashes, merge-squashes, and sync-merges (and amends, of course) do not perturb this fundamental diff.  The command “git mergebranch” (with three dots) is stable even if master advances or branch is rebased. The line numbers might change, and hunks might be rearranged, but the fundamental diff itself does not change unless there’s a conflict resolution (or an evil merge).  Atlassian’s Auto Unapprove plugin explores this in detail in issue #15.

If we were writing “Optimistic Build Status Propagation” from scratch, generating canonicalized diffs would be a big headache. Fortunately, the “git patch-id” command already has this covered, with some extra help coming from its “–stable” option:

     –stable
Use a “stable” sum of hashes as the patch ID. With this option
reordering file diffs that make up a patch does not affect the ID.

Here are some examples using master and branch from the diagram (clone from here if you must!):

git diff master...branch | git patch-id --stable
790e0c0693c61e28fa1b3eea204bafe3946f5cba

If I synch-merge (I’m on branch):

git merge master -m 'merge'
git diff master...branch | git patch-id --stable
790e0c0693c61e28fa1b3eea204bafe3946f5cba

If I retreat and rebase:

git reset --hard origin/branch
git rebase master
git diff master...branch | git patch-id --stable
790e0c0693c61e28fa1b3eea204bafe3946f5cba

The patch-id doesn’t change!  This makes the command (triple-dot-diff piped into patch-id) perfect for determining when rebases and other common branch operations have not changed the underlying work sitting on the source branch. Since the underlying patch has not changed, one can optimistically presume the build will probably have the same result.

Conclusion

Fast-forward merges are great because they avoid pointless merges and keep the history clean. Requiring successful builds before merging is great because it prevents broken builds. But add these together, and you might find yourself in an infinite rebase fight!

Fortunately, you can use $(git diff TARGET...SOURCE | git patch-id) to stop the fighting.

If you’re on Bitbucket Server, install the PR-Booster add-on to deploy the fix instantly.

Otherwise, roll your own, and let me know when you do! 

(p.s. For those on Bitbucket Server, I use Control Freak to enforce a fast-forward merge policy on git repositories I control.)

Do you still have doubts?

DOING GIT PULL WRONG

Too much fun with “git pull –rebase”

Note:  this article refers to “git pull -r” and “git pull –rebase” interchangeably. They are the same command, except the merge-preserving variation can only be specified via the long form: git pull --rebase=preserve

Introduction

I’ve long known that “git pull –rebase” reconciles the local branch correctly against upstream amends, rebases, and reorderings. The official “git rebase” documentation attests to this:

‘Note that any commits in HEAD which introduce the same textual changes as a commit in HEAD..<upstream> are omitted (i.e., a patch already accepted upstream with a different commit message or timestamp will be skipped).’

Thanks to the git patch-id command it’s easy to imagine how this mechanism might work. Take two commits, look at their patch-ids, and if they’re the same, drop the local one.

But what about squashes and other force-pushes where git patch-id won’t help? What does “git pull -r” do in those cases? I created a series of synthetic force-pushes to find out. I tried squashes, merge-squashes, dropped commits, merge-base adjustments, and all sorts of other force-push craziness.

I was unable to confuse “git pull –rebase,” no matter how hard I tried. It’s bulletproof, as far as I can tell.

Investigating ‘git pull –rebase’

The context here is not a master branch that’s advancing. The context is a feature branch that two people are working in parallel, where either person might force-push at any time. Something like this:

Git pull starting state
The starting state. Evangeline and Gabriel are working together on branch ‘feature’. Note: ‘evangeline/feature’ is actually Evangeline’s local ‘feature’ branch, and ‘gabriel/feature’ is Gabriel’s local ‘feature’ branch. Recreate it using the script below.

git init
echo 'a' > a; git add .; git commit -m 'a'
echo 'b' > b; git add .; git commit -m 'b'
echo 'c' > c; git add .; git commit -m 'c'
git checkout -b feature HEAD~1
echo 'd' > d; git add .; git commit -m 'd'
echo 'e' > e; git add .; git commit -m 'e'
echo 'f' > f; git add .; git commit -m 'f'
git checkout -b gabriel/feature
echo 'gf' > gf; git add .; git commit -m 'gf' --author='gabriel@mergebase.com'
git checkout -b evangeline/feature HEAD~1
echo 'ef' > ef; git add .; git commit -m 'ef'evangeline@mergebase.com'
git push --mirror [url-to-an-empty-git-repo]

The Experiment

In each scenario, Evangeline rewrites the history of origin/feature with a force-push of some kind, usually incorporating her own ‘ef‘ commit into her push. Meanwhile, Gabriel has already made his own ‘gf‘ commit to his local feature branch. For each scenario, we want to see if Gabriel can use “git pull –rebase” to correctly reconcile his work (his ‘gf‘ commit) against Evangeline’s most recent push.

Preconditions

  1. We assume Gabriel has correctly set up remote tracking for his local feature branch. This is a reasonable assumption since git sets this up by default when a user first types “git checkout feature”.
  2. We only tested Git v2.14.1 and Git v1.7.2 for this experiment. Perhaps “git pull –rebase” behaves differently in other versions.
  3. Important: we only use “git pull –rebase” (or -r).  Some people claim “git fetch; git rebase origin/master” is equivalent to “git pull -r”, but it isn’t.

Force-Push Scenarios

For each scenario, we are on Gabriel’s local branch feature. The graph on the left shows both the state of origin/feature (thanks to Evangeline’s force-push), as well as the state of Gabriel’s local feature and how it relates to Evangeline’s force-push.  The graph on the right shows the result of Gabriel typing “git pull -r.”

A scenario is deemed successful if “git pull -r” results in Gabriel’s gf‘ commit sitting on top of origin/feature.  Since Gabriel does not push back in these scenarios, his ‘gf‘ commit remains confined to his local feature branch.

  1. origin/feature rebased (against origin/master)
    This is the canonical example of why we prefer “git pull -r”.  The rebase notices that older commits ‘d‘, ‘e‘, and ‘f‘ on Gabriel’s feature branch are patch-identical to the rebased ones on origin/feature, and thus it only replays the final ‘gf’ commit.
    DOING GIT PULL WRONGDOING GIT PULL WRONGDOING GIT PULL WRONG


    Result: Success!

  2. origin/feature squash-merged (with origin/master)
    This is the rebase + squash combo meal.  Evangeline takes all work on feature, squashes it down to a single commit, and rebases it on top of origin/master.  She probably did this via “git merge –squash.” I did not expect “git pull -r” to be able to handle this, but I was wrong.
    DOING GIT PULL WRONGDOING GIT PULL WRONGDOING GIT PULL WRONG



    Result: Success!

  3. origin/feature squashed in-place
    This is the classic squash. Evangeline types “git rebase –interactive origin/master”.  In the interactive screen she marks the first commit as “pick” and every other commit as “squash” or “fixup”.  This squashes feature down to a single commit, but leaves the merge-base alone (commit ‘b‘ in this case). I also did not expect “git pull -r” to handle this one, but I was wrong here, too.
    DOING GIT PULL WRONGDOING GIT PULL WRONGDOING GIT PULL WRONG


    Result: Success!

  4. origin/feature dropped a commit
    For some reason Evangeline decided she wanted to drop commit ‘e‘ from origin/feature. She ran “git rebase –interactive origin/master” and marked every commit as “pick,” except commit ‘e‘, which she marked with “drop”.  I expected “git pull -r” to erroneously bring commit ‘e‘ back.  I was wrong.  Running “git rebase” instead of “git pull -r” did bring commit ‘e‘ back, and so there is obviously some deeper intelligence inside “git pull -r” enabling the correct behaviour here.
    DOING GIT PULL WRONGDOING GIT PULL WRONGDOING GIT PULL WRONG


    Result: Success!

  5. origin/feature lost their mind
    I have no idea what Evangeline was trying to do here.  If you look closely, you’ll see she reversed her branch (‘ef’ is now the oldest commit), she squashed the middle two commits, and she adjusted the merge-base so that origin/feature emerges from commit ‘a‘ on the mainline instead of commit ‘b‘.  This is one serious force-push!  I had no idea what to expect here.  I certainly did not expect “git pull -r” to nail it, but it did.
    DOING GIT PULL WRONGDOING GIT PULL WRONGDOING GIT PULL WRONG


    Result: Success!

  6. origin/feature went back to how things were (undoes the rewrite)
    Evangeline, either through her reflog or her photographic memory, happened to remember that origin/feature previously pointed to commit ‘325a76a.  Here she force-pushes origin/feature back to ‘325a76a‘ to undo her push from scenario 5. The command to do that is useful to know:  “git push –force origin 325a76a:refs/heads/feature”. Staring in awe at how “git push -r” did the right thing for scenario 5, all I could do was continue to stare when it did the same here. (Note: Gabriel’s start-state here is scenario 5, not the original start-state).
    DOING GIT PULL WRONGDOING GIT PULL WRONGDOING GIT PULL WRONG


    Result: Success!

Conclusion: Time To Revise The Golden Rule

Supposedly, the golden rule of git is to never force-push public branches.

Of course I would never force-push against ‘master’ and ‘release/*’.  As a git admin, that’s always the first config I set for a new repo:  disallow all rewrites for ‘master’ and ‘release/*’.

But all public branches?  I find force-pushing feature branches incredibly useful.

Industry has arrived at a compromise: defer the rewrite to the final merge. Bitbucket, Gitlab, and Github now offer “rebase” and “squash” flavours of PR merge. But it’s a silly compromise, because the golden rule itself is silly.  Instead of building complex merge machinery to dance around the golden rule, I think we’d be better served by reworking the rule itself. Three reasons:

  1. Force-pushes are useful!  Public amends, squashes, and rebases help us make better PR’s for code review.
  2. What is the actual point of the golden rule?  Are we trying to prevent lost work on the mainlines (e.g., ‘master’ and ‘release/*’)?  If that’s the point, then we’re much better off setting appropriate branch permissions on our central git server for those branches.
  3. Is the point to prevent the spaghetti graphs caused by default “git pull” behaviour?  In that case, a better golden rule would be never use default “git pull” and always use “git pull –rebase”, since it avoids spaghetti graphs while allowing history rewrites.

I propose a new golden git rule (in haiku form):

We never force-push master
or release. But always,
for all branches: git pull -r

Alternatively, you can make “git pull -r” the default behaviour:

git config --global pull.rebase true

Git graphs in this article were generated using the Bit-Booster – Rebase Squash Amend plugin for Bitbucket Server.

Want to learn more? Get in touch with us!

Discover More from MergeBase

Open Source Protection

Stay on top of the real risk of open source at any time.

Avoid false positives and get sophisticated upgrade guidance based on risk, compatibility and popularity.

More on Continuous Protection

Add RunTime Protection

Detect and defend against known-vulnerabilities at runtime. The only SCA to do so.

The quickest way to respond to an imminent threat like log4j with CVE-2021-44228.

More on Run-time Protection

Shift Left Now

CodeGreen is an early-warning defence for your in-house development and integrates directly into GitHub and BitBucket

More on BitBucket and Github apps