Keith Stobie, Microsoft
Over testing can actually be as bad for you as under testing. Whether it is testing to your own lofty expectations (versus those of the project), verifying too much in stress or verifying too much in one test, you might end up with a less useful result than other approaches. You must also be careful of under testing sequence and state or aspect of the software that are just hard to verify.
This experience report relates parables of mistakes I made and what I learned that you should apply to avoid them. These lessons learned are also cross-indexed as examples for parts of the book: Lessons Learned in Software Testing (Kaner, Bach, and Pettichord).
- Verify Stress tests out of band.
- Test Oracle size
- Test in the presence of failures.
- You must understand the goals of your release.
- Under testing sequence and state.
- Skipping coverage of the hard stuff
- Over and under testing combinations.
The many times I’ve taught various aspects of software testing, I find many students learn best by hearing stories that demonstrate a principle in action. I’ve been testing for so many years, I’ve gathered a number of scars that I attempt to help newer converts to the discipline avoid. Many of my lessons learned occurred before a book called Lessons Learned in Software Testing (Kaner 2001) was ever published, but in reading the book I could see how my experiences related to those in the book and correlate them as lesson #. The majority of the lessons are from my mistakes around over testing!
You can test too much in one test case causing unnecessary blocking due to bugs. You can test too much for a release by exceeding its requirements. You can verify too much during stress testing resulting in less stress testing or overly expensive stress testing.
Testing is a business activity with a cost. Using your resources wisely to most effectively cover the risks requires:
- Understanding the product and release goals,
- Dealing with common issues like known errors,
- Anticipating likely errors
- Choosing the right evaluation methods
2010 Technical Paper, Keith Stobie, Abstract, Paper, Slides
