Signals-based Testing of Microsoft Office with an AI-Driven Robot Army

I led and drove an effort to use an AI-driven robot army to test Microsoft Office. The goal was to achieve as much coverage of the application as we could via self-learning AI agents. Using the same analytical techniques as we used to look for failures and issues from regular customer usage, we further employed data analysis and machine learning to mine the product signals to find issues from the synthetic workload either before users would see them, or to get in-lab repro of issues product teams had been trying to investigate in the field.

• How self-learning AI agents can teach themselves how to generate application coverage
  
• How to use signals-based testing to analyze complex workloads
  
• Some of the problems that come of automation you do not control
  
• The challenges of covering very large product functionality with randomly driven workloads
  

Wayne Roseberry

Wayne Roseberry is a software engineer, tester, test lead, and test manager who worked at Microsoft from 1990 until 2023. He has work on such projects as MSN, Microsoft Commercial Internet Serer, Site Server, SharePoint, and Microsoft Office.