Felix Eu, Chin Pei Tan – Intel
Abstract: The risk-based validation approach is useful in multiple software program releases which share the same validation resources but with increasing test scope. Risk-based validation helps optimizing test coverage which will maintain the efficiency of the test coverage and ensure the right validation scope is implemented.
Risk-based validation is an approach adopted for a project which utilizes the impact analysis done on software code changes, prioritizing tasks, and taking informed risks to perform the validation. The software development team can provide the summary of software code changes in terms of impact analysis information based on the source code delta from the previous software release cycle. The impact can be categorized as low, medium, high, or any numbering format and presented in a table that describes the relation of the software code changes between the software feature and specific function/area in a project. The impact assessment information could be discussed in the project meeting and be assessed and analyzed by the software validation team to determine the validation strategy used for test optimization. In addition to the software code change, incorporating defect prediction outcomes into the impact analysis will greatly enhance the test optimization through a Risk-based validation approach. However, the risks taken should be balanced up with the Return of Investment (ROI) gained in a project.
In a critical situation when a software release timeline has been promised to the customer, it is crucial to ensure the entire software code is thoroughly validated. However, it is time-consuming to conduct regression testing every time there’s a small change in code to ensure the functional integrity of the software.
The risks in developing the validation strategy can be lowered down by taking the Risk-based validation approach with proper analysis and assessment of what is being changed in the software code instead of repeatedly performing the full regression testing. A risk-based validation approach will prioritize the tests based on the impact assessment from the overall software code changes. For example, software code change in dockers may impact any features that have a dependency on dockers image. In this situation, it can be decided that more validation shall be performed on dockers-related features over other features. Both software development and validation teams play critical roles to ensure the accuracy of the software impact given (input) and the tasks defined from the impact analysis (output).
To enhance further, the validation team may consider incorporating the defect prediction outcome to check whether the defect arrival rate is on-par with the defect prediction number. Depending on the actual defect arrival rate, if it is below prediction, the software validation team can adjust the validation strategy to improve the test coverage and vice versa.
Product risk level can be leveraged via Risk-based validation in the following ways:
– Start the Risk-based validation as early as possible to identify the correctness of the assessed impact and to determine if more validation shall be performed.
– Prioritize validation in the high-impact areas.
– Having mitigation and contingency plan in place to complete validation after uncovering high impact defects. For example, the workarounds to continue the validation until all impact areas are covered.
– Measurement of how well the Risk-based validation approach at finding and removing defects in critical areas.
– Proactive continuous risk assessment on non-validation areas to ensure they are defect-free throughout the release cycles.
The world of validation is rapidly changed, and the trend shows that it is transforming from traditional validation to risk-based validation. The maturity of the validation process in an organization is the key determination for this transformation. Many large-scale organizations have taken precedence. The process for this transition may include but is not limited to the following:
– Identify critical data and processes
– Risk identification and assessment
– Risk control and mitigation
– Risk communication and action
– Identify defect prediction model
– Developing a monitoring and control plan
– Build up Subject Matter Experts (SME)
There are challenges that arise during the transformation and the insistency pay-off all the efforts invested for this change. The efficiency and effectiveness to convert the impact analysis information into the validation tasks is the key success of the Risk-based validation approach.
Generally, risk should be properly tracked throughout the software product life cycle (SPLC). It should not be only limited to software code changes but should also include any aspects that may create threat to the project. The analysis of the impact-based risk assessment is critical to the success of the project commitment and minimizing escape defects to the fields. The ultimate goal to perform the Risk-based validation is to achieve a project outcome that balances risks with quality, features, budget and schedule for the time, resources and efforts that have been invested in a program.
Felix Eu, 2021 Technical Presentation, Paper, Presentation
