Monsters & Magicians: Testing the Illusions of Generative AI
From ancient myths of golems to science fiction tales of robots, humanity has long imagined autonomous machines that think and act like us. Recent breakthroughs in generative AI have moved these visions from our collective imagination into our everyday reality, in ways both magical and monstrous. As investors pour money into this rapidly evolving field and society grapples with its impact on our lives and livelihoods, software testers are left with a daunting question: How do we test software that produces inexplicable fuzzy outputs?
Join Ben as he ventures into the realm of AI illusions, exploring why familiar testing methods often fall short and what approaches can help test for the context-dependent risks of non-deterministic AI systems that are sometimes wrong.
Gain insights into:
- Recognizing and assessing AI-related risks
- Implementing test strategies tailored for the unpredictability of AI outputs
- Developing meaningful benchmarks to test capability and safety
Learn strategies to responsibly test generative AI systems while maintaining a balance between rapid innovation, safety, and regulatory compliance.
Ben Simo
Ben Simo is a context-driven software quality leader with over 30 years of experience dedicated to making software better. He champions a people-centered approach, leveraging automation to accelerate and extend testers' capabilities. Ben applies his automation-empowered, human-focused philosophy to help teams deliver better software that betters people’s lives.
His extensive background spans diverse roles, including analyst, developer, tester, test manager, product manager, toolsmith, educator, and quality coach. He has applied these skills across multiple industries including healthcare, finance, defense, education, marketing, e-commerce, and cloud infrastructure services. This breadth of hands-on experience has taught Ben to adapt quickly, effectively connect with diverse teams, and solve complex quality challenges.
Ben gained recognition for identifying critical issues during the launch of Healthcare.gov, demonstrating his investigative skills and deep commitment to quality. Known for his engaging and approachable style, Ben frequently shares practical insights and real-world experiences to help software professionals deliver meaningful results.
Currently, Ben practices his craft as a Principal Product Researcher at Tricentis, exploring innovative approaches at the intersection of AI and testing. His work focuses on identifying effective strategies for testing AI-driven systems and leveraging AI to empower testers to more effectively and efficiently test their products.