Recently, PNSQC reached out to 2019 Keynote Jason Arbon to get some insights into machine learning, tips on shifting company mindset towards quality, and more. Arbon is the CEO at test.ai, where the mission is to test all the world’s apps. In addition to authoring How Google Tests Software and App Quality: Secrets for Agile App Teams, he also previously worked on several large-scale products, including web search at Google and Bing, browsers Chrome and Internet Explorer, and operating systems such as Windows and ChromeOS.
PNSQC: What’s the difference between learning how to use machine learning (ML) for testing, and developing ML tools for testing?
JA: The key difference in building ML tools is that they need to be abstracted from any given product implementation — which makes things much more difficult. Using ML for testing is simply looking at the data you have and figuring out which type of ML/AI algorithm might be applicable. Developing ML tools for testing basically means building a mini-ML training platform, focused on testing problems, that supports the ability to customize training data, generate and validate models, and then apply them to testing arbitrary applications. One is easy, one is very much not easy. 🙂
PNSQC: It seems like AI and ML are cutting edge technologies. Maybe they work for some companies, but do smaller software development shops need to pay attention (yet)?
JA: AI and ML are ironically “old” technologies. That aside, I think smaller software companies have the most to gain from an AI-first approach to software testing. They don’t have the legacy code or sunk emotional costs of classic test case frameworks, etc. More importantly, they don’t have the resources to add hundreds or thousands of test cases or the time to maintain them, and this is where AI-first approaches shine. Most importantly, small companies need to move fast, and be “agile” — AI and data-driven approaches are far better in dynamic engineering environments vs classical testing approaches. Small companies can and should move AI-first in both product features and testing. Let the large corporations enjoy their hard-coded Java and custom test case frameworks and languages. Let the smaller companies leverage the latest technologies.
PNSQC: Do testers really need to become Machine Learning Engineers? Just as chefs don’t design stoves, wouldn’t testers work better at developing tests over designing the system?
JA: Agreed, too many testers worry they need to be machine learning engineers in this new world order. There will be two sorts of testers in the near future:
- Some super nerdy testers will be building AI-based testing infrastructure — but there won’t be many of those since AI means these tools are far more reusable than classic functional frameworks.
- And there will be testers who guide the machines, who have a “feeling” for what the AI is capable of doing and giving the machine examples or high-level test case definitions, and just let the AI do all the boring work of creating a model of the application, creating variations of the test cases, and execute them on behalf of the testers.
Ironically, I think most every tester will need to be less technical, but be far better at test case design. In the cooking analogy, testers will become customers at the restaurant, picking from the menu and asking for minor customizations — they aren’t toiling over the stove, nor building them.
PNSQC: Previous PNSQC Invited Speaker Michael Bolton recently tweeted about a presentation of yours discussing the ethics of AI — how testers and developers need to develop a critical mindset when using machine learning algorithms. Can you provide your views on this topic?
JA: Michael and I have chatted about this for a few hours. Michael is right to be concerned, and he’s influenced how I present things, with this context in mind. I also recommend some of his “critical thinking” methods, because testing AI bias is all about questioning everything — even questioning the bias and assumptions in the questions.
Bias is a real problem with machine learning — not just ethically, but also just for practical, functional, and business reasons. Bias is a large, unsolved problem today. I encourage folks to not just be “concerned” and metaphorically pull fire alarms but also understand how ML works, and dig into this bias problem from a technical perspective. Frankly, most folks only realize one-third of the bias problem, and that’s enough to scare them.
The reality is that there will always be “bias” — bias in the training set is what makes it useful, versus random noise. It is turtles all the way down… Bias is a great area for testers to explore; I just hope they do it constructively, versus throwing tomatoes.
PNSQC: The title of your upcoming Keynote presentation at PNSQC suggests that the team sets the quality level. Many managers might wonder how this can scale across an organization?
JA: Quality is bottom up. Individual people have to care. Management’s job is to make sure individual people care either through bribery, shame, inspiration, or via TPS reports.
PNSQC: What are the top 2 or 3 actions that you would recommend to shift your organization’s culture to focusing more on quality?
JA: My recommendations:
- Give everyone a monetary, monthly bonus based on quality metrics — everyone. People work to earn money, and they follow the cheese. Monthly to keep quality top of mind, and have the ability to recover from bad metrics/disasters.
- Make sure everyone knows how to test and look at production analytics. Most teams just never take the time. The Google Chrome team once took two days off, and everyone went through test creation workshops and couldn’t return to work until they’d created test cases, they were reviewed, they were checked in, and they were running.
PNSQC: What kind of robots do you have in your house and do you trust them?
JA: Two Roombas. One for upstairs, one for downstairs. Oh, and the AI testing bots that labor away on my MacBook Pro — thousands of little AI brains in fact.
Thanks, Jason! Stay tuned for his keynote presentation and other talks by software quality professionals at PNSQC 2019 this October 14-16. And get the latest info on workshops and discounts on Our Registration Page.
