PNSQC recently interviewed 2019 Invited Speaker Jan Jaap Cannegieter to get his thoughts on automation, testing tools, and more. With more than 20 years of experience in ICT, Jan Jaap is now Principal Consultant at Squerist, a consultancy company of 100 employees specializing in process management and testing. For a deeper dive, be sure to sign up for his upcoming conference workshop, “Test Management in Agile”.
“Do you want to test it quickly? Manual! Do you want to test it continuously? Automated! And keep experimenting and find out what works in your situation.”
All about Automation
Question: In your upcoming talk “The Dark Side of Test Automation,” you pose the paradox that more automation may cause a decrease in test quality. Can you elaborate on this?
Jan Jaap Cannegieter (JJC): We have seen this phenomenon in many disciplines. In aviation, we see more use of autopiloting and systems that intervene in crisis situations, but at the same time, we see that pilots don’t know what to do in case the computer fails. With devastating consequences.
We also saw this in the carpet weaving industry in the Netherlands in the 19th century. Carpet weaving used to be a very well mastered craft in the Netherlands. With the introduction of weaving machines, we saw a decrease in the weaving craftsmanship.
I see this happening the same with testing; many testers know everything about certain tools or programming languages, but they can’t tell me what they test and why they test it. And that is a bad thing.
Q: For organizations that are not yet sold on test automation, will your talk scare them or sell them on the benefits?
JJC: I don’t say Test Automation is bad, not at all! But I say we lose sight of why we test in the first place and how you do proper testing. I think we test to minimize risk or to gain information. And if tools help you do this in a more efficient or effective way you should surely use tools! A few weeks ago I saw a user-tester manually test all links on a website. I use web crawlers for the same test activity, that is much more efficient.
Q: In automation, it often seems that, when there is a major change to the system, a lot of changes ensue. How do you avoid throwing away a vast amount of effort in previous work?
JJC: This is something you can’t avoid all the time. What you can do is not start too early with test automation and judge upfront how much the system will change in the future. When you expect there will be big changes in the future you probably should not make a big automated regression test set. But making predictions is difficult, especially when we are talking about the future.
Something else you should do is structure your regression test set in order to make impact analysis easier and make it possible to determine where changes affect your test set. Still, if a business wants big changes in the system you maybe will have to rebuild your test set.
Tools, Analysis, and Always Exploring
Q: Do you stick with the same tools from project to project? How do you choose the right tools?
JJC: Theoretically you should always judge the situation and find the best tool for that particular situation! In practice, it doesn’t work this way. In my last assignment, I wanted to use Jmeter for performance testing, just because I know Jmeter very well. We did a proof of concept and it worked fine! Then I found out there was this organizational wide policy that Visual Studio should be used, and there is a performance tool in Visual Studio. At first, I didn’t like that, but looking back it worked very well, maybe in this situation even better than Jmeter.
I’m a member of the Test Automation Guild within Squerist, the consultancy company where I work. We often discuss projects we do, then a consultant explains his or her situation and we discuss which tool of tools are best in that situation. And discuss do’s and don’ts in that situation. This way the solution is not limited by the knowledge of one consultant. I think that is gold, pure gold. I learn a lot from it and our services to the customer improve.
Q: If you had to hire dogs or cats as quality analysts, which breed would you choose and why?
JJC: I’m not sure if I understand the question correctly, but I would hire cats. Cats have more a will of their own, are stubborn and are explorers of their own, which are good skills for testers. A drawback of cats is that they sleep two-thirds of the time, and you can’t manage or control them.
Q: In projects, you have worked on, who decides on the ROI for test automation? Is this a simple process?
JJC: ROI is too simple a process to measure the benefits; don’t use it. Please Google “Gil Zilberfeld ROI” — he has got some great blogs about it. We were never able to measure ROI of test automation — we are not able to measure it today and will never be able to measure it.
Compare two totally different situations: When you have a great automated regression test set, in terms of easy to run, high coverage and stable, you will probably run it every day, or every build. If you don’t have that automated regression test set, for instance, because the system isn’t stable enough, you will probably execute a very carefully designed set of manual test cases. Once or twice, not more often.
The number of test cases ran in these two situations is incomparable, you would never run the number of test cases manually you will run in your great automated regression test set. And when you only execute a number of test cases once or twice you would never automate it. I think the actual question should be: Do you want to test it quickly? Manual! Do you want to test it continuously? Automated! Keep experimenting and find out what works in your situation.
Q: Do you have any hobbies that reflect your tendency for testing? Does automation sometimes overflow into your personal life?
JJC: Being a tester (and business analyst) reflect on my personality, I think. A very important skill for a tester is critical thinking. I do that a lot in my personal life as well.
I’ll be an explorer all my life! Always trying new things, just to find out how it works. Probably you will find me walking around Portland the day before or after PNSQC. Why? Because I’ve never been to Portland before. What am I looking for? Don’t know yet, but I can tell you afterward. I think that is the testing mindset.
