Courtney Gray
In a thought-provoking fireside chat, Leapwork Co-Founder, Claus Topholt, shared his observations on the transformative impact of agentic AI on test automation.
As businesses are increasingly relying upon AI-driven solutions, the role of these intelligent and self-directed systems is becoming more crucial than ever.
Read on to discover Claus’s opinion on how agentic AI will not only enhance automation efficiency but also empower teams to focus on strategic innovation rather than repetitive testing tasks. You’ll learn how Leapwork is shaping the future of test automation and why agentic AI is set to redefine test automation industry standards.
Mind the agentic AI accuracy gap
Agentic AI is the next leap in the ongoing AI journey. With 85% of organizations having integrated AI applications into their tech stacks in the past year, the technology needs to work. However, 68% of those have already encountered performance, accuracy, and reliability issues (source: Leapwork’s AI and Software Quality Report).
In the fireside chat, Claus spoke about the potential future of using natural language AI to improve software testing so that all a tester needs to do is state their intent to get the desired result, especially when performing repeatable tasks such as continuous regression testing. In fact, he predicts a future in which testers only need to know their end goal: the AI will do the rest for them.
But as it currently stands, if a tester tells an LLM what they want to test and then they test it 10,000 times, hundreds, if not thousands of random errors could occur because the technology is not 100% accurate.
How accurate are large language models?
While LLMs do perform well on specific benchmarks, their real-world accuracy and overall reliability plummet from over 90% to around 60-70%. Claus states that this overall benchmark is currently considered “state of the art”. It is also similar for the buzzy technology of the moment: Agentic AI.
Ultimately, however, the accuracy gap is closing. He sees agentic technology as poised to revolutionize software development and testing, emphasizing that agentic AI “will transform the way we work and the way we think about software and testing. But the technology is forward-facing and a future thing. It is not in a state today that we can put it into Leapwork's stable category and indicate that it is 100% reliable within our platform".
Once the technology does reach this level of accuracy, we can achieve this. In fact, this forward momentum is “precisely why we should not abandon agentic AI but instead invest further in refining its capabilities”, to ensure it becomes a truly transformative tool for the future.
Leapwork is aligned with these advancements in AI, utilizing the technology and building solutions to bridge the reliability gaps. We do so by using various means including Leapworks' composable visual language alongside human and AI validation of AI-generated outputs.
Agentic AI and Leapwork: the bright future of test automation
While on the topic of agentic AI, Claus explained how Leapwork’s use of it is pushing the boundaries of how users can interact with software.
How? Through live conversations with an LLM about applications directly visible on the screen. This platform integrates computer vision, DOM analysis, vector database analysis, and LLM prompting to create an interactive AI capable of engaging with any type of software.
Crucially, Claus showed how this process operates with user oversight. The AI consistently seeks confirmation that it is on the right path and makes appropriate decisions based on the user's reply.
By breaking problems into smaller substeps, the AI executes one step at a time, continuously evaluating its progress. If an error occurs, it backtracks, reassesses its approach, and updates its understanding of the problem before proceeding.
Ultimately, agentic AI will revolutionize the testing process. What used to be hundreds of complex interactions between a tester and the software, will now allow for test generation from only a dozen or so prompts. This permission-based methodology will ensure reliability, making it possible to convert the processes into repeatable flows within the platform.
Leapwork Innovation Lab: autonomous testing you can trust
Leapwork delivers several AI-based solutions including AI Validate, the first software testing solution to help validate the accuracy and reliability of AI applications like Microsoft Copilot. In fact, Leapwork co authored this white paper to help businesses start with best practices in mind.
The future of AI in software testing is vast and bright. But the current accuracy gap when using LLMs needs to be bridged before businesses can trust autonomous AI to test critical business processes.
To address these challenges while experimenting with agentic AI, Leapwork has established an internal Innovation Lab dedicated to exploring the future of software test automation. Claus discussed how the Innovation Lab is a way for the company to exchange ideas and shape the future of software test automation.
The ambition of the Innovation Lab is to:
- Inspire the industry with cutting-edge ideas
- Build prototypes and experiment with new approaches
- Use AI where it makes sense while relying on stable, well-known algorithms where possible
- Showcase what the future might involve
Related blog: Testing of AI Applications with AI-Augmented Tools
Some of these features from the Innovation Lab may even become part of the Leapwork product.
In fact, this is how several of Leapwork’s building blocks originated, especially those using LLM such as AI Generate, as well as the AI Locator feature, which uses AI to locate elements on a page.
AI Generate
Leapwork’s AI Generate building block leverages the power of large language models (LLMs) to create synthetic, yet highly realistic test data. This enables teams to simulate real-world scenarios with ease, making testing more comprehensive and reflective of actual user behavior. With AI Generate, you can quickly produce varied datasets that enhance test coverage and improve the reliability of your automation.
Leapwork's AI Generate Block
AI Locator
Leapwork’s experimental AI Locator feature uses AI combined with on-screen metadata to intelligently identify and verify UI elements, like buttons, logos, or icons, ensuring they appear exactly as expected. By going beyond traditional selectors, AI Locator adds a smart layer of resilience to your automation, helping you catch visual or structural changes before they cause test failures.
Leapwork's AI Locator Feature
Leapwork’s AI Vision Building Block
AI Vision started as an experiment within Leapwork’s Innovation Lab and has recently become a featured innovation within our AI strategy. AI Vision will clearly be labelled as “experimental” in the platform.
So how does this building block work? Claus demoed AI Vision, showing how it helps to validate and extract specific data from an image or a specific screen. As an example, you could take a bank statement and ask for a specific element, such as the account balance or account number. He demonstrated how a user is able to generate a response in a few short seconds, which is validated by using image and text recognition that is run through a LLM.
Claus emphasized the use case of this building block is for situations where getting hold of the original structural data may not be possible, but using computer vision alongside a large language model could extract results.
AI Vision combines:
- Image and text recognition powered by LLMs (Large Language Models)
- Computer vision to analyze software interfaces where direct data access is unavailable
- Live AI interactions to validate test scenarios and flag errors proactively
Leapwork's AI Vision Block
Q&A Highlights on AI
At the end of his fireside chat, Claus held a brief Q&A on AI. Here is a brief recap of what was answered.
- How does Leapwork handle AI-driven testing, including chatbot automation?
Leapwork’s platform features the AI Validate Block, allowing users to compare AI-generated responses with expected outputs, such as conversations from Generative AI, ensuring chatbots and AI applications function as intended.
- What’s next beyond AI and ChatGPT?
Claus envisions a future where software requirements can be converted into structured models. By breaking down automation into manageable components, AI can assist where it excels, rather than being forced to handle everything. For example, instead of asking an LLM exclusively about a problem, you can chop up the problem and solve, for example, 85% of it with known, stable algorithms, and then you can use LLM to assist for the rest.
- What are you most excited about regarding AI and automation?
What excites Claus most about AI and automation, especially as a self-professed "product and visual language guy", is to solve problems with visualizations that any business user can relate to. AI-driven automation also enhances productivity, making software testing more intuitive and efficient, reducing manual or cumbersome workloads. He is also excited about the prospects of AI giving users a recap of a certain test flow, telling you what happened, what went right and where the issues occur.
Final Thoughts
Leapwork’s AI-driven, no-code test automation ensures businesses can focus on what needs to be tested, rather than on how to automate it. By utilizing Agentic AI and LLMs within the platform, Leapwork will continuously ensure quality, ultimately driving business agility and innovation. We want to redefine how businesses automate software testing, making it more accessible, efficient, and future-proof.