No one can say PNSQC folks aren’t up for a challenge! At this year’s conference, we asked people to form teams and come up with a solution to the following SQA Challenge:
Automated tests are often accused of being flaky. Sometimes they work, sometimes they don’t. Sometimes they report false failures. Sometimes the problem is with the product being tested and sometimes the problem is with the automated test or the test environment.
So, what can be done to improve the reliability and trustworthiness of automated tests, and efficiently and effectively address automated test flakiness when it does occur?
Your solution to this challenge should be a collection of specific practices, tools, technologies and approaches that produce automated tests that run reliably and produce trustworthy results.
Nine teams stepped up to the challenge, submitting entries that included highly technical guidelines for automated test development to recommendations for how to manage the people involved in testing and development. “All the responses were great,” said Program Committee member Shauna Gonzales “Reading through them, it was obvious that everyone had brought a lot of experience to the table.”
“It really came down to who did the best job of solving the problem that was asked,” said Program Committee Chair Tim Farley. “Some teams focused on just the testing part, but left out processes to support testing. Some focused just on detecting problems with automated tests, not preventing them.”
“The challenge was created so that people with all kinds of backgrounds could contribute to the solution,” added Ms. Gonzales. “And that’s the kind of broad answer we were looking for.”
Submissions were judged on a variety of categories, including whether they addressed trustworthiness and reliability, and covered mitigations and contingencies for automated test failures. Points were also given for solutions that were scalable and appropriate for large and small companies, and whether they relied on technologies available to everyone.
“The scores for the top 3 were very close,” said Ms. Gonzales. “We were counting quarter points,” added Mr. Farley. “The answers were all that good.” In the end, the team led by Alice Nodelman submitted the winning entry.
What can be done to 1) improve the reliability and trustworthiness of automated tests and 2) efficiently and effectively address automated test flakiness when it does occur.
To improve the reliability of automated tests consider environment, efficiency and cultural changes. When it comes to environment do checks to ensure that the automation system is in a testable state (services are up and healthy, necessary packages are installed) and work from a baseline/mocked data set with known inputs and expected and desired outputs. A healthy and known environment will result in fewer flakes. Select for more efficient use of test time – make sure that you are testing at the correct level. Don’t create a full stack test where a unit test is sufficient. With fewer tests in the system there are fewer opportunities for breakage. Finally, a cultural shift in the product development is necessary. Products need to be developed with automation in mind – providing unique indexing of resources and thorough command line interfaces. Many flaky tests are written trying to work around difficult to test products – make it a priority to build for successful testing.
Mitigation of flaky tests involves instant re-production, providing means to cheaply flag tests and, finally, further cultural changes to encourage ownership of testing results. If a flaky test occurs in automation it should be immediately retested (possibly more than once). Having reproduction on the same environment will help for diagnosis (three reds in a row = real red, red/green/red = look at the product/automation). If a test is shown to be flaky it should be taggable/flaggable to be removed from subsequent test suite execution – don’t clutter up your results with known, tracked issues. Finally, make cultural changes that encourage ownership of testing. Red automation should result in action from both product and QA; both teams should be invested in generating trusted results. Flakes should not be allowed to linger in the system to hide actual issues. A system that goes red and is ignored is a waste of resources.
Take these steps to build a better, more reliable automated testing system.
In addition to Alice, the team included Chris Cowell, Doug Messick, Julie Peterson, Purusottam Patel, Travis Stevens, Krismy Alfaro, Toby Dallum, and Cyril VanDyke.
“Having nine teams participate in this challenge and all submit such great, workable solutions, it’s clear that people took this seriously and wanted to share what they knew with their peers,” said Mr. Farley. “Next year, we should have more prizes,” suggested Ms. Gonzales.
Congratulations to all the participants in this year’s challenge. Hope to see you next year at PNSQC 2016
