Right-Sized Regression Testing to Reduce Cycle Time
In an ideal world, testers would run 100 percent of tests for every change in an application. However, in multi-year testing engagements where time constraints and customer budgets come into play, that’s not always possible. Check out these three steps for scaling regression times down by running the right steps at the right times.
Break down tests in the proper way
It can be difficult and time consuming to choose what to test, but most tools will have a way to break down tests and organize them into folders or suites. Even if using something as simple as an Excel chart, you can break up your test cases into tabs.
The question then is, how should test cases be broken down? Answer: Into functional areas of the application. The best way is by area of the software interface.
If you need to run tests on different levels of the application, such as the API level, include tests for the area of the software interface that the API interacts with. The first rule is to only go to two sub-folders or sub-suites. If you need to go deeper than this the feature would likely be better divided into more top-level folders containing more test cases.
The second rule is to make sure your sub-folders or test suites contain at least five test cases. Many suites/sub-suites or folders/sub-folders with few test cases in each may be even more difficult to work with than one folder or suite containing all the test cases since you would be adding additional layers and not breaking things into functions.
Manage the actual test cases
In a test repository, the addition of new features will continue to grow the number of tests to be executed. To keep tests to a reasonable amount, ensure that all test cases are current by removing test cases that are no longer valid parts of the application and editing test cases with new information. This information would be communicated in newly worked-on user stories or epics, in most cases.
Next, pare down the number of tests after initial executions. When executing new functions, you run extensive tests that can last a few cycles after the initial implementation. At a point soon after the initial implementation, cut out some of the more detail-oriented test cases. It is best to leave in test cases where there are failures in the application.
Determine what functional areas have changed
It’s key to have a good relationship with the development team in order to know what has changed. If you don’t have a relationship with a developer, you don’t have code access. And code coverage tools will determine what areas of the application have been touched. Those folders and suites would be run in the regression.
Choosing which folders to include in regression will require, and build upon, the use of the previous two steps. Code access for teams across multiple organizations is more of an exception rather than the norm. How would one make the determinations on what folders or suites to run? The answer is to have a deeper relationship with the developers.
This starts with making sure that the developers know what tests are in the regression. The developers will also need to know how the tests are broken down; but note, the whole development team does not need to do this. If one developer with an understanding of what is being changed in each build knows this information, this issue is covered.
The next step is to make a code coverage worksheet. This will consist of a checklist of all folders or suites, the coverage needed in those suites by percentage, and a place to keep any information regarding data requirements. This living document is built upon as new features are added to the application. For each new testable build, developers fill this out to shape the regression test. There is a level of trust that is implied between the developers and testers for this approach.
Regression tests are more manageable when you break down test cases in the proper way, manage the test cases, and determine what test cases to run. The process is not magic, though, and will not solve a lack of resources or automation when testing an large-scope application. These steps, however, will help in a situation where there is a time crunch and only so many tests can be executed in a time frame.