The Problem: AppSec Keeping Up with the Speed of DevOps
Over the past decade, DevOps has become prevalent across software development organizations. John Allspaw’s 2009 Velocity presentation entitled “10 Deploys per Day: Dev and Ops Cooperation at Flickr” made waves, with a deploy frequency that was seemingly unheard of at the time.
Today, however, multiple deploys per day is the norm. Born-in-the-cloud companies have DevOps baked in from the beginning, and legacy enterprises are focused on digital transformation.
As software delivery has accelerated, however, security has struggled to keep up. Application security teams have often been forced to choose between being a blocker to deploy to production or accept the risk of vulnerabilities in prod while they wait for their next review.
Modern security engineers know that they cannot be a blocker for fast-moving engineering teams. Instead, they must pave a road that ensures security is baked in through the development process. One approach to this has been heavy investment in developer training. I would argue, however, that good automation tooling in CI/CD is far and away the best path to ensure secure application delivery.
Developer Training: Not the Right Solution
One solution that has gained significant traction is developer training programs, where developers are put through security training to ensure that applications are built with security in mind from the beginning.
These programs take different models, whether it be training all individuals or train the trainer programs. All of them, however, rely on engineers retaining knowledge from a brief training for a long period of time. Additionally, these trainings are often focused on sample applications that are quite different from the applications the developers are working on. After a training is completed, engineers jump right back into building features and there is ample opportunity for overlooking something they just learned.
Said another way, can anyone find a developer who has never introduced a bug? Despite years of training and practice, even the most seasoned engineers still introduce bugs. Vulnerabilities (a.k.a. security bugs) are no different.
This is not to say that these training programs are bad, but there has to be a better model than broad-swath security training, which relies on knowledge retention, catching the edge cases, and never making mistakes.
Security Automation: Checking for Vulns on Every PR
Instead of relying on training programs as a primary means of delivering secure applications, engineering teams should use a tried and true methodology — automated testing in CI/CD. Test automation has become widely accepted, with unit tests and integration tests baked into the DevOps pipeline. Security should be no different.
Every pull request should kick off automated security tests to ensure that no new vulnerabilities have been introduced. There are lots of reasons to do this (finding vulns in production is too late, smaller units of change = much faster fix times, etc.), ultimately resulting in more efficient processes and more secure software.
If a new vulnerability is found in the pipeline, the developer who just wrote that code can quickly review and make a triage decision — should this be fixed now, be prioritized among other engineering work, or does it present such low risk that it is not worth fixing? Oftentimes, the fixes are simple and the developer who was just working on that part of the codebase can quickly address the issue, regardless of severity.
Automation is good both for finding vulnerabilities and for fixing them before they hit production. Engineering orgs are best off if they focus on strong automation instead of counting on human knowledge retention.
Automation + Training: A Complete Program
To be clear, I am still a big fan of developer training programs. Ensuring that security is baked in from the beginning adds a lot of value to organizations. After all, automation cannot possibly catch everything! But these programs should be paired with strong application security automation, with training viewed as the extra layer of security.
Additionally, as an industry, we should apply test driven development principles to security, particularly for items that cannot be caught with standardized automation tooling. Tenancy checks, for example, should be baked into the feature delivery. This is one example of skills that a good training program will instill in attendees.
Getting Started with AppSec Automation
Application security automation sounds like a heavy lift to implement, which sometimes leads teams to not even start down the path. In reality, however, widely available tooling makes it relatively simple to start adding AppSec checks in the pipeline.
I often encourage people to implement two types of security testing, which most teams can typically implement in a matter of hours:
Software Composition Analysis (SCA): SCA tooling scans the open source dependencies in use in your application and matches against known vulnerabilities. If there are vulnerabilities in a library you are using, you will be prompted to update the library, with many tools automatically opening a pull request for your review.
Dynamic Application Security Testing (DAST): DAST tooling runs active security tests against a running version of your application (or the underlying microservices) and APIs. This type of testing not only catches many vulnerabilities that would have been introduced via open source usage, but also catches vulnerabilities that your team may have introduced. Historically this category has relied on manual or scheduled scans of production, but I’m strongly biased toward tooling that catches vulnerabilities before they are in prod.
The important thing here is to just get started. Block a couple of hours, pick one of your services or applications, grab your beverage of choice, and instrument some basic application security automation.
