Adil Gheewala, Sam Bedekar, & Weifeng Yao, Microsoft
The test team in an organization often has a large set of new and regression tests to execute to ensure the quality of the features. The ever growing test matrix is another strain on the test organization as it needs to support multiple legacy clients, release after release.
Testing in production where real users test real scenarios is a great way to harness the power of the crowd, get a more end user centric test coverage, and reduce some of the tests the actual tester needs to perform. End users would install the application and use it in their normal mode. The TiP system collects the relevant user interactions and allows the owner to make decisions based on the data – feature test coverage and/or feedback on the feature.
The Lync Telemetry v-team has developed a tool called Scenario Compass which listens for events from Lync and any other Office 2013 applications to distinctively identify the set of actions/features a user exercises while using them. This allows the feature area owner to have data behind his crowd sourced tests. Scenario Compass also collects machine and application specific data which can be used to check off configurations in the test matrix. This allows the test team to focus their efforts on configurations with low coverages. Scenario Compass also flags failing cases. If a certain anticipated event does not occur in a given amount of time, that scenario is flagged as a failure. Using the failure analysis component of Scenario Compass, one could also do performance tests in the real world.
Scenario Compass has been used in the Lync client team to optimize the test passes by identifying areas covered by internal Microsoft users and reducing effort in them and identifying holes in coverage/configurations and focusing more in them. We also have built games into the tool to increase adoption and more importantly influence users to execute a certain set of features.
This paper, describes some of the challenges we faced in our testing, how we overcame them, the solution we built and how others can leverage those in their environment.
Adil Gheewala, Sam Bedekar, & Weifeng Yao, 2012 Technical Paper, Abstract, Paper, Slides, Notes, Video
