Sudhindra Kembhavi, McAfee
The intent of regression testing is to ensure that changes made to software, such as adding new features or modifying existing features, have not adversely affected features that should not change. Regression testing is usually performed by running some, or all, of the test cases created to test previous versions of the software (Wong 1997). Many risk-based techniques have been developed to select regression tests.
Complete regression testing is not always possible due to time and resource constraints. This may increase the risk of defects escaping to the field, which can be costly to correct. Arbitrarily reducing regression testing is not an acceptable solution either. Manual selection of the regression tests can be error-prone too by relying too heavily on people picking the tests.
Algorithmic selection of regression tests based on the association of test cases to lines of source code can be equally effective to manual selection of test cases in detecting regression failures, and can often do so with a smaller set of tests. Our process leveraged a commercial off the shelf code coverage analyzer with an in-house developed application to save coverage results for each test run. Our process for algorithmically generating regression test suites included:
- Base line creation
- Collecting code coverage information for each test case
- Creating a database of coverage information for each test case
- Regression tests selection
- Analyzing the source code changes for each build or check-in from the repository
- Selecting the test cases for each build or check-in
- Executing the selected tests to qualify the build
This approach significantly reduced the regression testing efforts for the project without reducing software quality. The regression tests selected by the algorithm uncovered the same defects that would have been found using manual regression test cycle planning. The reduced regression testing provided quicker feedback on the build which in turn helped keep builds stable and keep developers productive.
Target Audience: Intermediate
2015 Technical Paper, Sudhindra Kembhavi, Paper, Slides, Notes, Video.
