Sam Bedekar, Microsoft
Automation has its place in the test lifecycle, but when it comes to automation, the definition of a test case becomes a gray area. Is a test case an XML blob or other metadata that is interpreted by a harness; or is it scripted code (such as JavaScript or VBScript); or is it a function or set of functions within compiled code written in a language like C# or Java? This paper explores and summarizes the lessons we learned through experimentation with multiple methods while shipping three versions of Office Communicator. We learned the unique advantages that could be built into a custom scripting language, as well as the strengths leveraged by using advanced features of C#.
We will introduce the Simple Language for Scenarios (SLS). In SLS, we constrained users into writing test cases in a particular text-based format (not XML based). This reduced coding-style flexibility, but gave us a single paradigm of writing test cases and a point of injection via the interpreter. We used this point to achieve several goals – we were able to automatically marshal calls between different thread models, adapt existing cases for API parameter testing and enable security fuzzing without requiring each tester to have to write code to do so. For string injection we wrote an API string testing class where the script engine could automatically call each method in a loop and pass in all possible parameters within the context of an existing case simply by annotating the “start” and “end” points in the test case where the engine needed to loop. Additionally, with API testing, we were able to run a case to a particular point, and then bring up a UI to allow ad hoc testing. The steps executed by the tester in the UI would then be written back as script language that could then be incorporated into the test case library.
Ultimately, we landed with a hybrid model that managed to leverage the strengths of both approaches. Along the journey, we faced several facets to consider including development time for a test case, reducing cost of failure analysis, adaptability/extensibility of a test case to new environments and more. This paper will go into the technical details of our findings and how other software projects could benefit from our lessons-learned to more effectively deploy automated test cases.
2010 Technical Paper, Sam Bedekar, Abstract, Paper, Slides
