Shifting Left for Agile
The Agile Manifesto (2001) doesn’t mention testing directly, but one of its primary effects was a sea change in the practice of software quality assurance. Placing a premium on “responding to change” and “delivering working software frequently” required finding ways to collapse the formerly sequential processes of coding and testing. Agile testers strive to design the tests at the same time the developers begin coding. To minimize the time needed to validate changes, the tests should be ready when the code is, and they should be largely automated. “Shift-left testing,” as Larry Smith defined it in the same year as the Agile Manifesto, integrates test and development activities by involving QA early, getting tests into code as soon as possible, and looking for ways to speed up the release cycle.
“Shift Left” gets its name from the notion of a sequential, left-to-right software development process that typically proceeds from gathering requirements to design, implementation, test, and release. Moving left on the process line means moving to an earlier stage in the process.
The benefits of shift-left testing are many. First, in keeping with Agile precepts, it helps companies deliver business value more often and more reliably. Second, rapid incremental changes contribute to software quality and stability because small, continual changes are easier to manage than larger, less frequent, and more disruptive changes. Third, as has been shown repeatedly, shift-left testing reduces costs by finding bugs earlier. A NIST report from 2001 says fixing bugs in production costs businesses 30 times as much as fixing them early in development.
Benefits of Shifting Left
Deliver business value quickly and often
Improve software quality
Save money by fixing bugs early
Shifting Left for DevOps
In light of the Agile imperative to ship code early and often, the operational steps for code release began to seem like an obstacle. Getting finished code to production is a process in itself, traditionally owned by an operations team whose expertise includes managing servers, provisioning databases, protecting private keys, monitoring systems, and other work necessary for making finished code work in a live environment. Might there be a way to shift one step further left?
In 2009, motivated by a desire to overcome the friction that commonly hinders siloed ops and dev teams from releasing smoothly and reliably, Patrick Debois founded the DevOpsDays conference. DevOps aims to make operational processes easy, quick, and reliable so that developers can do more of the work themselves. And once again, automation was the key to unlocking business value. DevOps calls for a delivery pipeline in the form of a tool chain that builds, configures, tests, and provisions deployable software units–generally packages, containers, or virtual machines. Automated tests are needed to confirm that the deployable units are built and function correctly.
Automation was the key to unlock business value.
Just as shifting left for Agile required breaking down silos and strengthening cooperation between developers and testers, so shifting left for DevOps requires building trust between operations and development teams. The disciplines remain distinct but the lines of responsibility blur across roles. Operations advises teams and creates guard rails to support the safe provisioning of deployable units on demand. Development teams then own their own software from conception all the way to production. Operations engineers, like testers, become guides on a cross-functional team rather than rulers of an isolated domain.
Shifting Left for Security
Security came late to the left-shifting game for several reasons. First, application security was a less well developed field then. The universe of known vulnerabilities was much smaller. The first release of the OWASP Top Ten Web App Risks didn’t arrive until 2003. Some of the earliest application vulnerability scanners appeared a few years later (such as Paros in 2005 and w3af in 2007). Second, security scanners often take a long time to run, produce a significant number of false positives, and require some degree of security expertise to interpret correctly. These characteristics do not fit well in an Agile pipeline where the job of automated testing is to provide prompt feedback in the form of a clear green-light indicator if the system under test is ready.
As Agile and DevOps cleared the path for code changes to flow quickly from concept to production, security testing inevitably stood out as an obstacle. And again the pattern for change involved the same elements as earlier shifts: improving automation capabilities and blurring lines between siloed groups. If interpreting test results and implementing or approving fixes requires the intervention of a security team, then Security becomes a bottleneck and an obstacle to business goals. Instead, security teams and tools must help developers make good decisions. People addressing this shift gather under the rubric of DevSecOps.
Developer-Centric Security Test Automation
The goal of much modern application security test automation, then, is to provide results where they are most needed: to developers as they write code. Test automation should lower costs and speed delivery by surfacing problems as soon as possible to the people who are in the best position to fix them.
Agile, DevOps, and DevSecOps each put more responsibility on development teams and rely on automation to help them succeed. All three are primarily cultural changes. Developers must assume some responsibility for quality, operations, and security. Experts must provide whatever guidance and tools will help developers succeed, trust them to be responsible, and help if anything goes wrong.
Technology too must support the change. Shifting everything left requires a well-engineered CI/CD pipeline to provide automation so teams can execute complex tasks reliably and consistently. The end users of security test automation tools are no longer security professionals but developers, and tool makers must adapt by empathizing and integrating with the way developers work.
Success Criteria for Automated Security Testing
Knowing that the goal of automated security testing is to bring vulnerability data to developers suggests the criteria for success. A good solution must provide:
Developer-Centric Workflow: Developers must be able to generate and consume test results as close as possible to their own coding work–preferably along with other acceptance test results, either in their IDE or in the build pipeline.
Quick and Accurate Results: Long tests don’t get run often, and neither do tests that fail for no good reason. Slowness and inaccuracy are common in legacy scanning tools that probe for thousands of known flaws. Modern tools provide configuration options for more precise targeting, or even automatically drop tests irrelevant to your application architecture. For faster and more actionable results, scan small units such as microservices and APIs individually.
Developer-Centric Reports: Early scanners produced reports intended for security teams. In DevSecOps, scanner reports go to developers. This shift calls for clearer language, helpful explanations, example code solutions, and as much detail as possible about how to reproduce the problem.
Broad Integration Capabilities: Scanners need to integrate not just with standard elements in the CI/CD tool chain but with other parts of the company workflow as well including, for example, issue ticketing systems and enterprise reporting tools such as a SIEM.
Pitfalls in Adopting Test Automation
For all its benefits, test automation is complex and, like any complex process, it can go wrong. Here are some problems to avoid.
Solving Test Automation with Tools and Not People
As the review of Agile, DevOps, and DevSecOps transformations suggests, successful shift-left automation must prioritize cultural change first. Automating Agile tests doesn’t help much if the developers and testers don’t cooperate to agree on tests early, or if testers don’t trust the tests developers write. Similarly, Security must trust developers to internalize and act on Security priorities themselves.
Forgetting That Test Automation Is Code Too
Test automation is code and needs the same architectural leadership as any well-crafted software: an informed choice of the right tools, consensus on goals and methods, and discipline in execution. It requires staff who analyze complex systems and make consistent decisions about what to automate, and who can implement automation with clear, maintainable code. As James Bach wrote in 1999, automated testing projects will fail “if the machinery of testing distracts you from the craft of testing.”
In Conclusion
Twenty years of software process advances following from the Agile Manifesto have consistently aimed to create software iteratively, quickly, and reliably. Quick iterations necessarily rely on test automation. By expanding our automation tools and capabilities, we can include system provisioning and security testing in the delivery pipeline. Success requires finding developer-centric tools and processes that give developers prompt and precise feedback with every build, or even every check-in.
For more about integrating developer-centric AppSec testing in your CI/CD pipeline, try this article next.
Further Reading
This article touched briefly on problems and practices that have generated discussion for decades. Here are a few landmarks and informative articles for readers who might want to know more about the history, current state, or best practices in test automation.
Introductions to Agile, DevOps, and DevSecOps
The Agile Manifesto and Twelve Principles (2001)
What Is DevOps? (New Relic, 2015)
What is DevSecOps? (DevSecOps.org, 2015)
Understanding the Differences Between Agile & DevSecOps – from a Business Perspective (US General Services Administration, no date)
Security Standards Requiring Vulnerability Scans
ISO 27002-2013 (14.2.8 “Testing of security functionality needs to be carried out during development.”)
NIST SP800-53r5 RA-5
PCI-DSS 6.1 and 6.6
Test Automation
Test Automation Snake Oil (James Bach, 1996)
Guidelines to Create a Robust Test Automation Framework (Alliance Global Services, 2009)
When Should a Test Be Automated (Brian Marick, 2000)
Test Early and Often (Microsoft, 2012)
DevOps: Are You Pushing Bugs to Your Clients Faster? (Wayne Ariola, 2015)
Current State of DevSecOps
Every Security Team Is a Software Team Now (BlackHat keynote, 2019)
