Going from DevOps to DevTestOps with Continuous Testing
Over the past year, there’s been a lot of chatter in the software world around continuous testing. It was a hot topic at the annual Quest conference taking place this past May and is something our team is building skills in daily—and for good reason. As market demands continue to increase, there is an increased need to deliver higher quality software faster, and continuous testing is a prime means for making this happen.
For this reason, I like to say that DevOps should have been called DevTestOps and CI/CD should have been called CI/CT/CD. In order to achieve that accelerated delivery, you need continuous testing in parallel with coding. Otherwise, you will be deploying poor quality software faster—and no one wants that!
A majority of Agile teams know the value of continuous testing, but there are still a number of questions that remain. And, while testing has significantly shifted left from the days of waterfall development lifecycle, continuous testing has not yet become commonplace in DevOps and CI/CD processes. Here, I break down some of the main questions we receive related to continuous testing.
What is continuous testing?
Continuous testing (CT) is the use of automated tests whenever possible within the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Whereas continuous testing needs test automation for feedback speed, continuous testing is not the same as automating all tests.
Continuous testing works because more frequent testing provides faster feedback on the state of the code. This removes the bottlenecks of waiting for an application to be tested prior to release because the application is being tested constantly throughout a sprint and not merely at the end of it. It also reinforces the ‘definition of done’ of a user story, including the completion of testing and not just development activities.
What type of tests should be included?
Tests are driven by what needs to be validated within the CI/CD process. Continuous testing embeds various types of software tests into the CI/CD process, including functional tests at multiple levels: unit tests, integration tests, API/services tests, and a finite number of UI tests, as well as non-functional tests: security and performance.
Each test has a single purpose and is performed at the lowest level possible. It’s typical to run more unit and API tests, which are stable, as compared to the brittle UI tests subject to constant business changes to the software.
Regardless of type, tests must run very quickly, be scalable, be reliable, and provide actionable feedback to developers, so as not to slow down the CI/CD process. This is why automation is extremely important with automated tests triggered within the CI/CD process.
How do I know if I have the right tests running as part of my continuous testing effort?
When it comes to testing, even continuous testing, more does not always mean better. It is important to continuously refactor test code to make sure it’s still relevant and to exclude redundant tests from test suites. There should not be over emphasis on the percentage of test automation and the test automation suite size. In the past, this approach created huge and often slow running end-to-end workflow regression tests with little insight to actual test coverage.
Today’s continuous testing requires small automated tests that run quickly and can be easily maintained. These granular tests built for the CI/CD process can then be strung together for workflow regression tests.
We recommend taking a risk-based approach to test automation and score the scenarios by a few different factors for prioritization, including:
Risk: For the test scenario, what is the probability of occurrence and severity of impact to customers or business operations, if there is a defect?
Value: Does the test provide actionable information? If it fails, how much time is required to fix it?
Cost Efficiency: How long does it take to develop the test and how easy is it to maintain the automated test compared to manual testing?
History of test: What is the value and stability of the test in terms of volume of executions, defects detected, and frequency of test breakage?
Any software change can require writing new tests, modifying existing test flows, retiring tests, splitting tests, or merging tests. Experiencing a failed test in the CI/CD process means it’s already too late to discover the software has changed.
Source code management (SCM) tools should be used, if possible, to identify changes to the software code base, and changes to the corresponding automated tests can be made. Since automated tests are also code, it is equally important to properly maintain test code with source code management tools and processes.
How can you tell if your continuous testing effort is succeeding?
It will take time to achieve continuous testing. You will know your approach to testing is succeeding if:
- You find defects prior to product owner demo/UAT or before going live in production.
- The ‘Definition of Done’ for a user story is that it has been both coded and tested.
When code passes the test, the user story is done. - You can complete your testing of user stories within the same Agile sprint as the developers instead of lagging a sprint behind.
- Your tests are reliable and run quickly within your CI/CD process; your tests are not flaky with a lot of false positives that require review.
- Testing is not a bottleneck to a deployment.
- Your CT test results are used for the readiness to deploy decision.
- Metrics show an improvement in your software’s time, cost and quality.
How do we help our people adapt and adopt a mindset that is more in line with continuous testing?
Culture change is a very important consideration. First, it’s imperative that business and IT leadership establish “quality” as a priority within the culture. Software quality should not be sacrificed to time and cost pressure. Testing must be seen as a valuable safeguard to mitigate risk of poor-quality software, and not purely a cost to be cut out (or reduced) in projects whenever possible for cost savings.
Testing must not be an afterthought and instead must be integrated within a project, starting with a test strategy for continuous testing. It’s important to allocate adequate testing hours to a project, otherwise it’s difficult to perform continuous testing.
And, moreover, testers should have an ‘equal seat at the table’ during all project activities. For example, testers must participate in sprint planning to review user stories for testability, determining types of tests (manual and automated), and estimating the testing effort. Scrum masters are instrumental in bringing testers ‘to the table.’
Developers must be willing to commit their code and integrate it more frequently (e.g. hourly, nightly), so that code is continuously available to test. Otherwise, it is impossible to shift testing left within an Agile sprint. There are many reasons developers delay in committing/deploying their code. One major reason is the lack of a solid automated CI/CD process to integrate and deploy code to various environments (DEV, QA, staging).
Lastly, you can’t shift left and keep up with the pace of development activities unless you have properly designed automated tests and corresponding test data that is created and consumed by the tests in a way to be reused. This leads to high confidence that when a test fails, it is due to an actual defect detected. Having highly skilled automation engineers as part of your Agile team is essential, whether you are using coded or scriptless automation. This includes expertise in the automation tool/framework, language, CI/CD process/tools, and test design.
Final thoughts: what’s next?
Remember, you can’t do everything at one time, but with small, proactive steps you can implement continuous testing and achieve your software delivery quality, cost, and time to marketplace goals.
One of our strategic partners, Tricentis, has developed a helpful 5-level maturing model for continuous testing that illustrates how companies can be at different levels of maturity for their continuous testing. The maturity model provides a roadmap for how to progress through the five levels along with the efficiencies gained at each maturity level. We apply this maturity model to evaluate the current state of continuous testing and to recommend a roadmap for rolling out continuous testing within your organization.
To get you headed down the CT road, we offer a complimentary one-day discovery session to explore how you can achieve continuous testing. For a discovery session or to receive more information on continuous testing, DevOps, and more, please contact us at info@spr.com.
