Cristina Manu, Pooja Nagpal, Donny Amalo, and Roy Patrick Tan, Microsoft
Code coverage is a good technique for understanding what code has been exercised by an existing test bed. It is often debated how much resources need to be invested in increasing the code coverage. This paper is a case study of the code coverage effort that we did during our testing cycle for a component of .NET Framework 4.
At the end of the test cycle we had a focused initiative to increase the block code coverage from 91% to 100% in order to measure the return on investment (ROI) of such an effort.
We calculated the ROI based on the number and importance of the issues found, the time invested, the increase in code coverage and bug yield in comparison with other test activities. Our results showed that it’s neither prohibitively expensive to achieve effective-100% code coverage nor did we find an exponential increase in the number of bugs with the higher code coverage, as suggested in some papers in the field. Although we did find some systemic holes in our test bed, they did not uncover any major issues in the product.
In this paper we discuss what parts of our development process (such as the propensity of testers to develop more complex scenarios first) may have contributed to this lack of bugs.
2010 Technical Paper, Cristina Manu, Pooja Nagpal, Donny Amalo, and Roy Patrick Tan, Abstract, Paper, Slides
