Winning with Flaky Test Automation

Wayne Roseberry, Microsoft

Flaky test automation drives everybody nuts. Run it once, it fails, run it again – nope! So you stop running those flaky tests, and everyone is happy until a nasty bug gets in there that those tests would have caught. It’s a trap. If you want to run faster and release more often, you cannot afford the wasted time from noisy flaky tests, but in order to go faster, you have to know about the bugs.

This paper will discuss how the Office team came to terms with this problem, embraced the reality that real bugs live under the flaky crust and figured out how to use those tests, and their flaky behavior, to their best