4 Steps Businesses Miss When Calculating the ROI of Test Automation

Test automation aids businesses by accelerating their digital transformation. However, without management support, the right tool, processes, and a quality mindset from the top-down, test automation won’t succeed. 

Part of the journey of getting support from management is to accurately measure the return of investment (ROI) of test automation. However, some businesses can miss steps or miscalculate. As a result, their automation efforts fall flat. 

In this post, we explain the common mistakes businesses make while measuring the ROI of test automation and provide a short guide that outlines what you should be measuring. 

Common mistakes businesses make when measuring test automation ROI

Whether you’re coming from a manual testing background, or you have an internal automation framework in use, you may have faced one or more of these challenges when calculating the ROI of test automation. 

So what are the common mistakes to avoid?

1. Not factoring in test automation implementation and maintenance costs

Most test automation can drive productivity. However, to realize the true value of test automation, it has to be implemented, built, and maintained. These costs aren’t always factored into the initial ROI calculation. 

There are: 

• Licensing costs. This applies if you are using a vendor as opposed to an in-house framework. 
• Training costs. While all automation requires training, some require more than others. If you are using a free automation framework, or licensed automation, the tool will require more training. This is because your team may need to learn how to script in order to build test cases. 
• Maintenance. Script-based automation requires significant effort and cost to maintain. This cost and effort will depend greatly on the number of assets that need to be automated, their complexity, and the type of tool chosen to automate tests. With a self-developed framework using open-sourced solutions like Selenium, resources aren’t freed up. Instead, they spend the majority of their time developing and maintaining script-based tests. 

The maintenance of test automation can be kept to a minimum, but it requires adopting best practices and a tool with minimal maintenance. We outline how codeless test automation can help you achieve this. 

2. Miscalculating the time needed for failure analysis in coded frameworks

When you have internally developed a framework for testing a functional UI, your test cases will be comprised of lines and lines of code. 

The bigger the application under test (AUT), the more lines of code there are. If something breaks, you have to find out where the test broke. You also have to identify whether the failure is due to a bug, or a change in the source code of an application. 

Parsing through these lines of code is a resource-intensive exercise, and it’s where most go wrong when creating their ROI measurement. It’s either forgotten about or miscalculated. This causes development teams to look for simpler automation solutions. 

3. Underestimating the costs of incidents

Another important factor that can be underestimated when measuring ROI is the cost of incidents caused by bugs leaking into production. The further downstream it is identified, the more expensive it is to solve. 

This is best illustrated with a test automation pyramid. 

• On the unit level. If a team of developers finds a bug on the unit level, they can fix the bug without involving many resources and send their unit of code downstream. 
• On the service level (integrations). Once the units of code are submitted to create a new feature, it is pushed to a test platform for QA to assess. If a bug is identified by QA, that team member has to understand what caused the bug. They have to gather enough evidence (screenshots and text) on the source of the bug and assign it to the right developer. The developer will run a couple of iterations to verify that it is a bug. If QA is stretched and under pressure, they may not be able to embrace enough testing, meaning that the bug would move downstream. 
• On the acceptance testing level (end-to-end testing). The bug now becomes part of the next stage where business users are tasked with testing the software. They need to provide information to QA and the developer. Both QA and the developer will go through the same process of gathering evidence and verifying that the issue is in fact a bug. 
• In production. If the bug makes it past the above quality gates to the production environment, the bug could be found by an end-user. The end-user will call the service desk. The service desk will be responsible for gathering evidence and sending it to QA. QA will go through the process of gathering evidence to send to the developers, and so on. On top of the additional resources consumed, the severity of the bug can cause business disruption, hindering the workforce from doing their job. 

4. Not communicating the benefits of test automation once implemented

A common misconception is that investment into a tool alone will be successful. However, it also requires the constant communication of the benefits. It may seem straightforward, but a number of projects are unable to scale because the benefits haven’t been clearly and regularly presented to executives. 

We cover this topic in more detail in our on-demand webinar with Head of Business Value Consulting, Kristian Bjørn, where you’ll learn everything you need to know about measuring the business value of test automation. 

What should you be measuring?

Stick with tangible measurements that have quantifiable outcomes on productivity and cost. While time to market and quality software are benefits of good test automation, they are difficult to measure.

These points may not be visible in the beginning of your automation project, but can be considered as additional benefits that are realized as your automation project matures.

So what can you measure before implementing automation to get a realistic idea of the cost and benefit of automation?

We detail these points in our guide to measuring the ROI of test automation for tangible results that you can communicate to management.