Test Automation:
The Ultimate Guide

Test automation or automated testing is software (separate from the software under test) that is used to control the execution of tests.

Automated testing frees up time and resources so that you can test faster, with higher accuracy and at a lower cost. 

Read more: What is Test Automation?

Gartner define automated testing as follows: 

Manual vs. Automated testing 

The use of test automation does not exclude manual testing. In fact, they compliment each other very well.

Automation is about assisting people in reaching their objectives in a smart way, not to replacing jobs.

Cars, assembly lines and calculators are just a few examples of technologies that have been introduced because they allowed people to work smarter, not harder.

By introducing automation in testing, testers aren’t replaced, but are relieved of tedious, repetitive work, so that they can instead focus on higher-value tasks where they use their skill set to design test cases that assure quality and increase testing coverage - both quantitatively and qualitatively.

Listen to this episode of Decoding Automation where Sune Engsig, Senior Evangelist at Leapwork, explains how governance, stable automation and having an 'automation mindset' as opposed to a 'manual mindset' are key requirements for successful and scalable automation.

Read more: Test Automation vs Manual Testing: 10 Considerations
Get the whitepaper: Manual Testing vs. Automated Testing

Types of testing

There are many different types of testing.  Some are better suited for automation than others.

There are two main approaches to testing

• Black box testing means testing an application without knowing its internal workings or implementation - like a user. As the name implies, the application is like a black box to the tester, who supplies input and examines output.
  
  
• White box testing (or “glass box testing”) is the opposite. The tester knows the internal workings and implementation details (code, configurations, etc.). Relying on this knowledge, the tester will test certain paths through the code.

Using one or both methods, testing is done at various stages of the software development life cycle to find bugs.

Another way to organize tests is via the testing pyramid.

The testing pyramid

The testing pyramid is designed to give an overview of the different levels of testing, from the smallest units to the overall connected processes. Level 1 is at the base of the pyramid, where white box testing takes place. Level 2 and 3 are usually approached with black box methods.

• Level 1: Unit testing means testing individual units of code or groups of units in a piece of software. Unit testing is only possible using the white box method because unit tests are written in code and are closely tied to software development. These tests are therefore also done by developers.
  
  
• Level 2: Integration testing means testing how units of code interact with one another. This involves testing a chain of components, sometimes external components too, that together handle a process or business transaction. Integration testing often includes testing the interactions between hardware and software as well as other infrastructural components.
  
  
• Level 3: End-to-end testing is, and the name implies, about testing a process from end to end. The scope of end-to-end testing depends on the process, but end-to-end tests often span across multiple technologies. The purpose of end-to-end tests is to ensure that a flow works as intended from the perspective of the user. For this reason, the black box method is often used.

Related reading: How to Execute an Agile Testing Pyramid with Automation

More types of testing

In addition to the types of testing included in the testing pyramid, you will also come across other frequently used terms for testing. 

• Functional testing is used to ensure that functionality specified as part of the software’s requirements works as intended from the end-user’s perspective.
  
  
• Regression testing can be described as “repeated functional testing” but shouldn't be confused with retesting. When new features are built, regression testing ensures that old features of the software continue to work as intended. As the product evolves, the number of regression tests grow too, making it one of the most time-consuming types of testing for large businesses. It is also one of the best candidates for automation. 
  
  
• Exploratory testing is an ad-hoc approach that tests a piece of software with little or no planning apart from the scope. The tester relies on their experience and knowledge of the typical pitfalls and business requirements. Exploratory testing is generally considered a great foundation for creating repetitive test tasks to be performed later. It uses the black box method. Due to the unpredictable nature of this method, it cannot be automated.
  
  
• Load testing (also called stress testing) means testing how software behaves under increasingly unfavorable conditions, such as large inflows of users measured in logins per second, or spikes in business transactions such as shopping basket operations. Although load testing falls under the category of black box testing, analyzing and understanding the results of a load test is highly complex and typically requires in-depth code and infrastructure knowledge.
  
  
• Performance testing means testing how different parts of software perform in terms of speed and effectiveness at processing required business transactions. Performance testing uses the black box method.
  
  
• Penetration testing is used to simulate attacks by hostile parties to evaluate the security of the software. For example, an e-commerce platform might use this method to ensure that hackers can’t access credit card numbers or other confidential data. It is typically performed using the black box method, but by highly skilled specialists with deep coding skills that attempt to “open up the black box” to exploit what is inside.
  
  
• Usability testing means testing a piece of software from an end-user’s perspective with a focus on user-friendliness, aesthetics, and navigational effectiveness. It uses the black box method.
  
  
• User acceptance testing (UAT) is one of the few types of tests not performed by the vendor or producer of the software being tested, but rather by the customer receiving it, or customer ambassadors. This type of testing is done to ensure that the software meets the requirements and works as expected from the customer’s perspective. It is typically used as the final gate before payment of the deliverables. UAT is performed using black box testing.

• UI testing means testing the user interface. The user interface is what the user sees when using an application. UI test automation tools are typically tools that record a user's interactions with the interface, and mimic those interactions automatically.
  
  
• API testing means testing the application programming interfaces (APIs) that are exposed by the developers to third-party developers. It is done to determine if the APIs meet the expectations for functionality, reliability, performance, and security. Although API testing requires code skills, it is typically performed using black box testing because only knowledge of the API’s input and output parameters is required.

Read more about the different kinds of testing types not covered in this guide “List of software testing types” by Software Testing Fundamentals.

Next are general guidelines on finding a good candidate for test automation.

Types of automated tests

A good starting point for figuring out if a process is qualified for automation is to see if it fits the following criteria:

• The test is highly repetitive and takes a long time to perform if it is done manually (e.g. regression testing)
  
  
• The testing path is predictable, meaning it has been proven to verify a process before through manual testing
  
  
• The test is prone to human error, potentially because it is highly repetitive and involves migrating large amounts of data

If the process fits all these criteria, it will in most cases be a good candidate for automation.

Businesses are at risk of disruption. As new business models emerge and customer demands keep evolving, enterprises everywhere are struggling to stay relevant. They must be ready to adapt the way they do business.

With technology being used in new and more complex ways, automation is being employed across enterprise systems, cloud-based commerce, and cross-channel user experiences. This has pushed quality assurance (QA) and testing to the front of the stage.

QA is more important than ever

With more features and customizations being made to software and enterprise applications, there are more opportunities for something to break. This has put QA and testing at the very core of digital transformation. They ensure that bugs are found in software before they reach end-users or cause critical systems to crash.

Every change to your system results in a growing test suite. Each change introduced has to be tested too. It’s a costly endeavor, with the inefficiency of testing activities being a major contributor to the rising costs. But without testing, the quality of your software is at risk.

Related reading: How to Facilitate QA in Software with Test Automation

The cost of doing nothing

Consider the following scenario. A vendor of software solutions to the financial sector wants to release updates to a web application multiple times per year. Before each release, the product must go through careful testing.

The test team, consisting of three full-time testers, has identified and mapped out 300 test cases. 250 of these test cases are regression tests. The remaining 50 tests are more loosely defined and exploratory in nature and will continue to be performed manually.

Designing, planning, and executing the 250 regression test cases is extremely time-consuming when done manually. On average, each test case requires one hour of manual testing efforts, amounting to approximately six weeks of testing. The 50 exploratory tests aside.

To meet a minimal coverage for a single release, the team of three testers must spend two full weeks designing and running their tests.

The company has six releases planned for the year ahead – one release every second month. This would require a total of 36 weeks of testing.

The 250 test cases for regression testing are the bare minimum required for an acceptable product quality. QA and product managers know they need more thorough testing to increase product quality and to stay competitive. But there’s no budget to hire more testers.

For teams experiencing an increasing pressure to do more with less, automation can be introduced to relieve testers from the repetitive, predictable, tedious tasks.

From the business perspective, there are three main drivers for automation: risk reduction, cost reduction and productivity.

These drivers for automation manifest themselves differently in organizations. Below are three examples of businesses in three different sections, who all see a need to automate their testing.

• An insurance company is threatened by increasing pressure from competitors and see that the market is not responding well to their online self-service application. To support better and more frequent releases of this web service, the company’s Digital Team implements test automation to test more frequently and more broadly. They are driven by the need to reduce risk and reduce costs.
  
  
• A software provider is spreading their tester resources very thin and must rely on developers to help test during critical sprints. To avoid allocating more costly and valuable developer hours to testing than necessary, the company automates regression testing. They are driven by the need to contain costs an increase productivity.
  
  
• A health care provider has experienced highly critical outages of clinical systems. To increase patient safety, the organization’s IT department automates monitoring of clinical-critical systems. They are driven by the need to reduce risk.

Benefits of test automation

There are many different frameworks and tools available for test automation. The outcome of automating tests will depend on how successful the implementation is, but in general, by automating tests, the following outcomes can be expected:

• Increased productivity: More tests can be performed with high accuracy once tests are automated.
  
  
• Faster feedback: When tests are performed faster and earlier in the Software Development Life Cycle (SDLC), feedback can also be given sooner to developers.
  
  
• Faster time-to-market: Faster test execution and continuous feedback loops mean the overall SDLC can be shortened and the frequency of releases can be increased.
  
  
• Cost efficiency: When resources are optimized, costs can be reduced.
  
  
• Increased agility and market responsiveness: Shorter release cycles allows businesses to better respond to changes and prioritize resources.
  
  
• Reduced risk of human error: Automation allows repetitive, predictable tests to be performed with high accuracy. The risk of human error is thereby minimized.
  
  
• Improved quality: More efficient testing means the scope and frequency of testing can be maximized.
  
  
• Higher job satisfaction: Testers can experience higher job satisfaction due to the elimination of highly repetitive tasks.

These positive outcomes are enhanced when no-code test automation is used, as implementation, adoption and execution become easier and quicker. We'll return to the benefits of no-code test automation later in this guide, but you can also learn about this in our blog post: 7 benefits of codeless test automation.

Limitation of test automation

It’s important to note that while automation has many benefits, it also has its limits; it only tests what you tell it to. Though this may seem obvious, it’s important to keep in mind because even after unit, integration, and performance tests have passed, a single end-user can make the whole system crash in seconds. This usually happens if the user does something the developers did not expect.

This is why human testers are so valuable. They try to anticipate and mimic the behavior of end-users. Testers act as the users' ambassadors in the software development process. In other words, testers perform tests on the users' behalf.

Related reading: 5 Failures in Test Automation - and Best Practices for Tackling Them

When it comes to choosing how you will automate your tests, there are many different options. These fall under three main categories: code-based frameworks, low-code tools, and no-code tools.

Learn more about each of these with the following resources:

• Test Automation Frameworks: Why They Fail
  
  
• Low-code vs. No-code Automation
  
  
• 9 Reasons to Use No-code Automation Tools

Based on the title of those resources, you can probably guess which type of automation we are advocating for. In the following, we will dig deeper into why we believe no-code automation is the way forward - even for teams with highly skilled developers on board.

The test automation paradox

The positive outcomes of test automation are clear. So why don’t all businesses automate their testing?

The answer can be found when taking a close look at most test automation tools.

So far, test automation tools have dictated that automating requires coding. Since test automation takes place within the IT department, you might argue that this doesn’t matter, because many people within IT are fully capable of programming.

There’s just one problem: the majority of testers aren’t programmers. They are business process experts. Code-based automation solutions prohibit them from contributing to their area of expertise: the quality of business processes.

So when test automation requires programming, it’s either pushed to the developers in the team, or it’s left with the tester to figure out and spend time on - either way, this skills gap ends up draining valuable resources and time, often creating bottlenecks, and costing more than could potentially be saved by replacing manual tasks with automation.

This is what we call the Test Automation Paradox - when setting up and maintaining test automation takes up more time and resources than what can be saved by automating.

No-code test automation is a game-changer for testers. A test automation tool that uses a visual language instead of code to describe tests is a much more efficient way for testers to work. Here are 7 reasons why.

The benefits of no-code test automation

• No-code lets testers match the pace of the development cycle, because it’s significantly easier and faster to set up.
  
  
• No-code lowers the maintenance burden and lets teams scale automation much more easily because it doesn’t require endless amounts of scripting.
  
  
• No-code frees up resources and enables better testing at lower costs because testers don’t have to spend their time on coding, or learning to code, and developers don’t need to learn how to automate. Testers can focus on testing, and developers can focus on developing.
  
  
• No-code allows teams to extend the scope of test automation, as it’s easy to scale and can work across platforms, as opposed to e.g. code-based Selenium.
  
  
• No-code improves adoption and collaboration across teams due to its ease of use and its visual language. Learn more about successfully adopting test automation. 
  
  
• No-code eases documentation and reporting processes, as tests are automatically logged and can be reviewed more easily thanks to the visual interface.
  
  
• No-code speeds up the release pipeline and supports CI/CD by allowing much more efficient testing, thanks to all the points above.

Learn more about no-code test automation in our blog post How No-code Test Automation Closes the Skills Gap in Software Development.Why use no-code if you can code?

No-code empowers those without coding skills to create and contribute to test automation. But what about those who can already code? What do they stand to gain from adopting a no-code solution?

1. Your time can be better spent elsewhere

When you're building a test automation framework, it has to be well structured for it to take minimal work to maintain. It can take hundreds of valuable developer hours to build. This takes you away from value-creating tasks like developing new features. It also costs the business more than it should. 

2. Code-based test automation doesn't scale

As a business grows, so too do the software requirements. New features are added, customizations are made. All of these changes require testing. But for code-based frameworks to keep up with the growing demand for testing, you need the resources to maintain the framework. The more it takes to maintain test automation, the less value it brings, and the more difficult it is to scale. 

3.  You'll probably inherit someone else's code

Taking over another developer's code isn't always worth it. Either, you dissect the code and find a way to make it work for you. Or, you scrap the project and start again. Both cases require a significant investment of time, which once again takes your time away from working on developing new features. 

4. No-code makes it easier to collaborate

A common language closes the skills gap between developers and non-developers. It enables testers, business users, and developers to use the same approach to build and maintain test automation. 

5. Maintenance is reduced with codeless automation 

With a code-based solution, you have to comb through lines and lines of code every time a test breaks, or an update has been made to your system under test. 

With a codeless solution, maintenance is simplified because the interface is easier to navigate. 

Read the full article: Why Should I Use Scriptless Test Automation if I Can Code?

The role of no-code test automation in Agile, DevOps, CI/CD and Continuous Testing

Agile, DevOps and CI/CD all go hand in hand and play a critical role in delivering quality at speed. Next is an overview of each of these methodologies, and a list of resources where you can learn more.

Agile

Agile is a methodology or approach to testing and development that aims to make the Software Delivery Life Cycle (SDLC) more flexible and effective. It’s made possible with tools such as test automation, which ensure continuous feedback from testing to development. Agile testing thereby makes businesses less vulnerable to changes in the market as they can adapt faster.

Get the Agile Testing whitepaper to learn how to work with the agile methodology in testing for successful software development.

DevOps

DevOps is the bridging of development and operations. It’s mostly used as a job title, but it’s also an approach. DevOps has emerged in connection with the agile movement with the need to increase the frequency and speed of product releases while maintaining or improving a certain standard of quality. DevOps encourages collaboration and communication between teams from development to deployment, and requires automation throughout the pipeline in order to enable the flexibility and continuity that is required to deliver quality at speed today.

Get the DevOps and Test Automation whitepaper to learn how to automate testing in the DevOps lifecycle.

CI/CD

CI/CD stands for Continuous Integration and Continuous Delivery/Deployment - and that is exactly what is - a continuous, rather than a siloed, waterfall approach to software development. The CI/CD pipeline, which is typically built by the DevOps, is a series of automated tools that work together to secure a constant, accurate flow of code from development to deployment.

Watch the webinar on continuous testing to learn how you can achieve continuous delivery with automated testing.

This guide has introduced a new paradigm in test automation: No-code.

No-code test automation lowers the barriers to building and executing automation, for both technical experts and business experts, and helps teams avoid the common pitfalls in implementing automation.

Listen to our podcast episode of Decoding Automation to brush up on why no-code test automation helps teams close the skills gap in software development and build maintainable and scalable test automation.

Next, we'll take you through important steps in getting started with test automation. This includes finding the right test automation tool as well as following best practices and guidelines that will help you rethink or optimize your existing approach.

Creating a test automation strategy

As a part of getting started with test automation, it's ideal to have a test automation strategy in place. Particularly when building test automation intended to be scaled at some point, it’s inadvisable to take an ad-hoc approach to automation.

Having a strategy in place does not counteract agility by putting rigid processes in place. Rather, it ensures that you take the most important factors for your specific business needs into consideration. This includes the size of your business, the industry it’s in, the technologies it uses, and the resources available.

We have created a test automation strategy checklist of factors to focus on which can be used to get started with test automation. It can also be used to optimize your current automation process, if you have already begun your automation journey.

When evaluating automation tools, go through the following checklist of requirements:

• Technical capabilities cover all your enterprise’s primary applications, e.g. Salesforce, SAP, Oracle, etc.
  
  
• Strong governance features, incl. authentication, access control, versioning, and complete audit trails.
  
  
• Jobs can be run both locally on desktop computers and remotely on virtual machines and in the cloud.
  
  
• Reporting and analytics capabilities and/or integrations to PowerBI, Tableau, or similar.
  
  
• Allows business users, who are not programmers, to visually design flows without having to write code.

Download the complete automation tool checklist here.

Measuring the business value of test automation

When laying the tracks for a test automation solution, it's important to consider what business value it brings.

You should start this process by asking three fundamental questions: 

1. What challenge/challenges are you addressing with test automation?

2. What is the desired state once test automation has been implemented?

3. What is the desired outcome of the investment, and how can it be measured? 

Being able to answer these questions and measure the value of test automation should be a core step in any investment you make. We've outlined more on this topic in our post on how to measure the business value of test automation. 

To help you further, we've put together a brief article on what businesses miss when calculating the ROI of test automation. 

From not factoring in implementation and maintenance costs, to underestimating the costs of incidents, we've outlined four common mistakes, and how to avoid them. 

In addition to putting a strategy in place to achieve success with automation, there are also a number of best practices that should be followed to achieve the desired outcomes.

Most mistakes in test automation are predictable and can be avoided by following the best practices for building maintainable and scalable test automation.

• Increase test coverage gradually with automation: Plan for gradually automating your test suite. We recommend starting out by focusing on the flows that are easiest to automate. In most cases, you will find that it’s the relatively simple and very repetitive flows that, by far, take up most of your testing time.
  
  
• Build isolated test cases: Start by building test cases that only test one thing. This way, there is no doubt about what goes wrong when a test case fails. Instead of bundling up multiple tests in one test case, it is best practice to build reusable components with your test automation tool. This way, it is easy to reuse the logic contained in other test cases, and the time required to create a new test case is minimized.
  
  
• Build automated test cases that are independent and self-contained: This way, they can all be scheduled at once to be executed anytime and in parallel, i.e. across different environments.
  
  
• Ensure collective ownership of test automation: Remember that success with automation is dependent on a team’s collective knowledge. Adopt a test automation platform that all testers can work with, so that automation becomes a natural part of the daily work of all team members.
  
  
• Use a tool with a good technical fit: Implementing test automation is a long term strategic choice and should be handled as such. When evaluating automation tools, look across your organization and identify all the applications and technologies that could be potential targets for automation. Identify the scenarios where test cases need to move between technologies, e.g. web and desktop applications, and select an automation platform that has matching capabilities.

Learn more about how to build maintainable and scalable test automation in our guide with best practices for building test automation.

Once you have an understanding of the best practices for building test automation, you'll need to decide how you want to create your test cases. There are essentially two ways to do this: with a test automation framework or with a codeless testing tool.

Which is the best solution for you and your team?

To answer that, let’s take a step back to talk about what a test (automation) case is, as seen from the perspective of the tester.

The elements of a test case:

• A name or other unique identifier
  
  
• A process that must be followed step-by-step
  
  
• A set of input (test data)
  
  
• A set of expected results
  
  
• A set of prerequisites for executing the test

All these elements are worth going into details about, but there’s one that is at the core of how we should approach test automation. It has to do with how to describe a step-by-step process.

A test case is essentially a description of a business process. For example, “log in to an application and check that the username is properly displayed,” or “put items in a shopping basket and check that the total including discount is correctly calculated”.

There's more to test automation than building test cases. Once you and your team have implemented a test automation tool and you've created a suite of automated tests that provide sufficient coverage for your quality assurance, you've reached the next stage in your automation journey.

There are a number of steps you can take to further improve your testing processes and increase the return on your investment in the test automation tool. Next, we'll take you through these steps.

Test automation lets you execute more tests, faster. This means you will also get more test results. So how can you handle those most effectively?

Setting your team up for success with test automation analysis is arguably just as important as achieving success with the implementation.

Here are four tips for analyzing test automation results:

1. Set up automated monitoring to make sure testers spend their time most effectively.
   
   
2. Figure out why test cases are failing by utilizing your test automation platform's logging, debugging, and reviewing functionalities. Learn more about test management and how to review automation flows in Leapwork. 
   
   
3. Integrate with your release management platform either by pushing or pulling test results.
   
   
4. Ensure fast and transparent feedback with shared dashboards of real-time test results.

Learn more about each of these steps in the factsheet: How to Analyze Test Automation Results

In extension of the best practices for analyzing test automation results outlined above, it is important to consider how to ensure environments that produce reliable test results. The quality of your test results is only as good as the quality of your test environments.

The main requirements for good test environments are:

1. Make the application in question testable by dealing with third-party system dependency. This can be done in one of two ways:

• Encapsulate the application for testing purposes, either by relying on mock data inside the application or by using a proxy service.
  
  
• Simply set up automated monitoring of e.g. a web page calling all relevant third-party services.

2. Make test environments fit your DevOps pipeline. This includes considerations related to load balancer, deployment, service accounts, environment installation, and more.

3. Decide how to manage test data. There are several ways to ensure that test data has a certain quality, including:

• Baselining from production data
  
  
• Bootstrapping the database
  
  
• Creating test cases which contain the data generation in themselves

Learn more about how to set up high quality test environments in our blog post: Why Good Test Environments are Crucial for Successful Automation

Artificial Intelligence (AI) is an intriguing – and sometimes intimidating – phenomenon. It is no longer the stuff of a faraway future.

The world of software quality assurance dreams of artificial intelligence. The promise of a scalable, self-improving, digital workforce is too good to ignore. And in theory, there are endless ways in which AI can be utilized in test automation.

But, to date, no one has developed a computer program of general intelligence able to think on its own. The more advanced applications of AI in quality assurance, e.g. robots designing test cases on their own, are still fiction.

What does exist are clever statistical data analysis methods. We might call these artificial intelligence algorithms. Some of them are machine learning (ML) algorithms that can build clustering and predictive models out of even small amounts of data. Other algorithms mimic human decision-making, such as the way humans interact with software. Image recognition, for example, imitates humans' visual cognitive processing of what is on a computer screen. Algorithms like these are incredibly powerful when applied in the right way.

Learn more about AI, ML and hyperautomation:

• Hyperautomation: What, Why and How?
  
  
• AI in Test Automation: Hype or Help?
  
  
• AI and Automation: A User Guide

Gartner define Robotic Process Automation (RPA) as follows:

"Robotic process automation (RPA) is a productivity tool that allows a user to configure one or more scripts (which some vendors refer to as “bots”) to activate specific keystrokes in an automated fashion. The result is that the bots can be used to mimic or emulate selected tasks (transaction steps) within an overall business or IT process. These may include manipulating data, passing data to and from different applications, triggering responses, or executing transactions. RPA uses a combination of user interface interaction and descriptor technologies. The scripts can overlay on one or more software applications."