Michael Millerick, Blend
Blends primary product is a customizable web application. We leverage a large suite of pre-merge headed end-to-end tests to ensure its continuing functionality and quality. As the number of engineers increased, most had little visibility into the cause of flakey tests. The flakey tests would fail builds, yet engineers had little motivation ability to remedy the cause of the failures. This tragedy of the commons created instability and prevented pull requests from merging in a timely fashion. This instability made it impossible to release as often as we desired.
We set out to improve stability without removing flakey tests from the test suite. We took a multifaceted approach to process and cultural improvement.
We addressed the negative feedback cycle of instability with cultural and engineering changes:
- We defined distributed ownership and analytics for tests to create visibility and accountability.
- We provided engineers with tools that removed the need to watch the status of tests on pull requests.
- We eliminated escape hatches that allowed engineers to bypass tests.
- We improved the scalability of our test infrastructure.
This approach improved the pass rate of tests and reduced the tribal knowledge and barriers to merging code. The improved ease of merging, combined with the infrastructure changes, improved the rate we released new features to production to almost daily. These changes cut the time engineers waited for tests to finish by half. They also reduced the amount of infrastructure per engineer required to run end-to-end tests.
This paper describes the implementation details and impact of each of the four points in the approach described above. It will also examine how the changes made for each point in the approach reinforced the need for the other points.
Key takeaways include:
- Cultural changes can be just as impactful as code changes.
- The process must evolve as an organization evolves.
- Tools that facilitate processes drive adoption of those processes.
- Analytics are key to pinpointing pain points in processes.
Michael Millerick, 2019 Technical Presentation, Paper, Slides
