Laura Bright, McAfee
End-to-end automation provides many benefits to developers and QA engineers including faster defect detection and greater product stability. Automation reduces testing overhead and continually monitors overall product quality, enabling developers to quickly identify and fix defects in both new features and regressions.
A challenge to the success of any automation framework is involving the engineers who benefit from this effort. Our experience from past projects has been that most engineers do not monitor automation runs closely due to time constraints or limited accessibility of results. Automation team members are left with the task of interpreting results and filing defects, which places a greater load on the automation team and increases the overall time required to find and fix defects.
This paper presents solutions we have introduced to close this gap in the automation cycle and involve all engineers in the automation process. We draw on experience building end-to-end automation frameworks for two projects and present specific examples of how our solutions have helped developers catch and fix their defects in a timely manner. Our solutions include the following:
- Automatic automation triggers – every time a new build is ready, the automated tests begin executing immediately.
- Email notification of results – An automatic email summary of test results is sent to all team members after every run, ensuring visibility and a prompt response.
- User-friendly UI for result reporting and analysis – Engineers do not need detailed knowledge of the underlying framework implementation to view results and identify the root cause of a defect. Once a defect is fixed and a new build is available, the automation suite will execute automatically and verify the fix.
- All engineers actively contribute to the framework through test script authoring, feedback, and maintenance.
To date, our automation has helped developers and QA catch 19% of the total defects in our current project. Our daily automation runs and reporting solutions have greatly increased developer and QA involvement in automation efforts and improved the stability of our weekly builds.
Laura Bright, 2012 Technical Paper, Abstract, Paper, Slides
