We recently caught up with upcoming 2019 Keynote Dionny Santiago to ask him about one of his passions, artificial intelligence, and his keynote, “Surviving The AI Testing Apocalypse.” The principal quality architect at Ultimate Software also notes that test automation is a primary area of interest; he is focused on research and development efforts to apply AI and machine learning (ML) to software testing. Dionny is also currently pursuing his Ph.D. in Computer Science at Florida International University.
PNSQC: Human-coded user interface (UI) automation tests seem to be very flaky; will AI testing be as difficult, and why?
DS: Overall, I think we’re headed towards code-less, AI-driven, self-healing test automation.
Currently coded UI automation involves constructing page objects that are coupled to the implementation details of a specific web application. Page objects often include references to DOM and stylesheet information for a specific SUT. One promising application of AI in testing is ML-based element selection. Neural networks (or other ML models) can be leveraged to raise the level of abstraction by which test automation interacts with web and mobile components.
While minor DOM/styling changes would likely break test scripts that leverage traditional DOM-based element selection, a sufficiently trained ML model can get very good at recognizing elements — much like humans recognize elements — allowing a test script that leverages ML-based element selection to continue execution in most cases.
PNSQC: How do you convince a Quality team to move to AI? How do you retain the good retrainable testers that feel threatened during the Apocalypse?
DS: Short answer? If you build it, they will come.
Longer answer? Like anything else, we have to show teams the value in a new approach before it has a chance of being adopted and of thriving thereafter. With AI and Testing, there is also bound to be quite a bit of resistance as some of us may be afraid of our jobs being completely automated. While the thought that our jobs might be significantly transformed can be intimidating, the amount of attention surrounding AI and Testing is very motivating, as I believe there is significant potential for AI to improve our quality of life as testers.
I think the key is to demonstrate to quality teams that as test automation evolves through the use of AI, with thousands and thousands of bots at our command doing the work we’ve taught them to do, our time will be freed from much of the burdens of test automation scripting and maintenance. This will allow us to focus on more important aspects of testing, such as exploratory testing and finding defects. As far as retaining good retrainable testers goes, I think testers will be intrigued and motivated by the new challenges that are waiting for us as we start testing systems that use AI.
PNSQC: How do you picture Siri and Alexa after the testing apocalypse? 😉
DS: Loose, out of control, and in need of testing.
Jokes aside, and at the risk of contradicting myself, I think there will still be testing after the testing apocalypse. It will just be quite different than what we’re used to today. Today, we put forth a great deal of time and effort in designing test cases and writing and maintaining test automation. All of that will start to change, and rapidly. But the key thing to remember is that AI and ML are also software, and they also need to be tested.
When the bots have taken over, our job will be to ensure they are behaving correctly. It’s basically the Oracle problem, squared. We’ll need to develop a lot of new techniques to test these new “intelligent” systems. In a nutshell, testing is not going anywhere.
PNSQC: In 2018 you presented a poster on Aeon, a semi-opinionated UI automation framework. Opinionated frameworks enforce designing tests the “right way,” limiting the options available to engineers. How is this different or better than an un-opinionated solution?
DS: Proponents of opinionated software believe that the design should only realize the “right ways” of doing things — limiting the options available to engineers. The Angular UI framework is a good example of an opinionated design framework. Building a UI with Angular requires following very specific patterns that have been deliberately put in place to force the resulting design to be more modularized and more testable.
The Aeon framework translates this concept into test automation. Aeon combines an opinionated internal domain-specific language with the page object pattern to enforce good test automation practices. The result is an approach that provides engineers with enough power to solve new automation challenges while promoting the development of highly stable, maintainable, debuggable tests.
PNSQC: Peering into the crystal ball, how soon do you see the average tester using AI-assistance to generate test cases? Really?
DS: We are seeing an increasing trend of companies and thought leaders sharing ideas that blend AI and testing. We’re also seeing test automation systems that incorporate some aspect of AI into their solutions. There are also already a few open source tools that allow anyone to start applying AI to testing in just a few minutes (https://github.com/UltimateSoftware/AGENT and https://github.com/testdotai/appium-classifier-plugin). These projects explore multiple applications of AI to testing, including ML-based element selection and test generation. While the intersection between AI and testing is still a relatively new space, it is growing at a rapid pace, and I believe in just under 5 years we will start to see more evolution and widespread adoption of these and other similar approaches.
Thank you Dionny! In addition to his keynote, Dionny will be leading the workshop “Augmenting Your Test Pyramid with Layered Front-End Testing.” Be sure to sign up for his workshop, and get the latest info on other workshops on our Registration page. We’ll see you at PNSQC 2019, coming to Portland October 14-16!
