Wayne Roseberry, Microsoft
One of the challenges with performance and stress load tests on servers is to construct patterns of usage and data that mimic what will happen in real-world deployments. Artificial load patterns are valuable and often efficient means to find flaws, but they fall short of determining if the product will behave as desired under customer expectations. Some playback solutions exist that allow for relatively simple load testing, but these are not well adapted to server software having complex, dynamic states required to construct valid operations and requests.
This paper describes how the SharePoint 2010 team solved this problem by creating a tool capable of sampling real world usage data and building a model capable of describing and building the load test necessary to simulate real-world traffic patterns. The tool was designed to build load tests that run using a shipped web-testing product, Microsoft Visual Studio Test System 2008, thus making the tests and data portable to any other environment where this product is available. The solution allowed us to incorporate new load patterns relatively quickly, both from in-house deployments and customer deployments of our server product. Adding such workloads to our test suites had previously been so expensive and difficult as to be non-feasible. With the new solution in place bug discovery rates increased as did fix ratios, as we knew that the load tests were based on expected traffic patterns. We were able to add pre-production validation testing to our internal deployment processes, providing costing and stability predictions, something that was not possible before. Finally, the test tools were released to market to allow customers to do their own capacity planning and management via a load test kit able to adapt to their own workload needs.
2010 Technical Paper, Wayne Roseberry, Abstract, Paper, Slides
