Tips and tricks Best practice guides, FAQ & more
In this test automation guide, we answer all your questions related to testing and test automation to equip you with the knowledge to effectively and strategically approach this field.
Whether you're looking to learn about best practices for testing, or you're working in a business to optimize and streamline software development, this guide will help.
We've included a navigation bar to help you find what you're looking for. You can read the guide from beginning to end, or jump to the sections you find relevant.
Let's start with the basics.
Test automation or automated testing is software (separate from the software under test) that is used to control the execution of tests.
Automated testing frees up time and resources so that you can test faster, with higher accuracy and at a lower cost.
The use of test automation does not exclude manual testing. In fact, they compliment each other very well.
Automation is about assisting people in reaching their objectives in a smart way, not to replacing jobs.
Cars, assembly lines and calculators are just a few examples of technologies that have been introduced because they allowed people to work smarter, not harder.
By introducing automation in testing, testers aren’t replaced, but are relieved of tedious, repetitive work, so that they can instead focus on higher-value tasks where they use their skill set to design test cases that assure quality and increase testing coverage - both quantitatively and qualitatively.
Listen to this episode of Decoding Automation where Sune Engsig, Senior Evangelist at Leapwork, explains how governance, stable automation and having an 'automation mindset' as opposed to a 'manual mindset' are key requirements for successful and scalable automation.
Read more: Test Automation vs Manual Testing: 10 Considerations
Get the whitepaper: Manual Testing vs. Automated Testing
There are many different types of testing. Some are better suited for automation than others.
Using one or both methods, testing is done at various stages of the software development life cycle to find bugs.
Another way to organize tests is via the testing pyramid.
The testing pyramid is designed to give an overview of the different levels of testing, from the smallest units to the overall connected processes. Level 1 is at the base of the pyramid, where white box testing takes place. Level 2 and 3 are usually approached with black box methods.
Related reading: How to Execute an Agile Testing Pyramid with Automation
In addition to the types of testing included in the testing pyramid, you will also come across other frequently used terms for testing.
Read more about the different kinds of testing types not covered in this guide “List of software testing types” by Software Testing Fundamentals.
Next are general guidelines on finding a good candidate for test automation.
A good starting point for figuring out if a process is qualified for automation is to see if it fits the following criteria:
If the process fits all these criteria, it will in most cases be a good candidate for automation.
Businesses are at risk of disruption. As new business models emerge and customer demands keep evolving, enterprises everywhere are struggling to stay relevant. They must be ready to adapt the way they do business.
With technology being used in new and more complex ways, automation is being employed across enterprise systems, cloud-based commerce, and cross-channel user experiences. This has pushed quality assurance (QA) and testing to the front of the stage.
With more features and customizations being made to software and enterprise applications, there are more opportunities for something to break. This has put QA and testing at the very core of digital transformation. They ensure that bugs are found in software before they reach end-users or cause critical systems to crash.
Every change to your system results in a growing test suite. Each change introduced has to be tested too. It’s a costly endeavor, with the inefficiency of testing activities being a major contributor to the rising costs. But without testing, the quality of your software is at risk.
Related reading: How to Facilitate QA in Software with Test Automation
Consider the following scenario. A vendor of software solutions to the financial sector wants to release updates to a web application multiple times per year. Before each release, the product must go through careful testing.
The test team, consisting of three full-time testers, has identified and mapped out 300 test cases. 250 of these test cases are regression tests. The remaining 50 tests are more loosely defined and exploratory in nature and will continue to be performed manually.
Designing, planning, and executing the 250 regression test cases is extremely time-consuming when done manually. On average, each test case requires one hour of manual testing efforts, amounting to approximately six weeks of testing. The 50 exploratory tests aside.
To meet a minimal coverage for a single release, the team of three testers must spend two full weeks designing and running their tests.
The company has six releases planned for the year ahead – one release every second month. This would require a total of 36 weeks of testing.
The 250 test cases for regression testing are the bare minimum required for an acceptable product quality. QA and product managers know they need more thorough testing to increase product quality and to stay competitive. But there’s no budget to hire more testers.
To meet minimal coverage for a single release, the team of three testers must spend two full weeks designing and running their tests.
With the current resources at their disposal, they have settled on a minimal level of sanity testing. This leaves the test team spending most of their time on regression testing or product maintenance at the expense of exploratory testing and product improvement.
This solution isn't sustainable. Over time, the number of regression tests will grow, and testers will be under even greater pressure, while time for innovation and growth remains stagnant. This forces the business to either turn up spending or miss out on valuable opportunities.
Either the QA team must grow, or the testers must become more productive. The cost of doing nothing is compromised product quality, and, consequently, a loss of market share.
For teams experiencing an increasing pressure to do more with less, automation can be introduced to relieve testers from the repetitive, predictable, tedious tasks.
From the business perspective, there are three main drivers for automation: risk reduction, cost reduction and productivity.
These drivers for automation manifest themselves differently in organizations. Below are three examples of businesses in three different sections, who all see a need to automate their testing.
There are many different frameworks and tools available for test automation. The outcome of automating tests will depend on how successful the implementation is, but in general, by automating tests, the following outcomes can be expected:
These positive outcomes are enhanced when no-code test automation is used, as implementation, adoption and execution become easier and quicker. We'll return to the benefits of no-code test automation later in this guide, but you can also learn about this in our blog post: 7 benefits of codeless test automation.
It’s important to note that while automation has many benefits, it also has its limits; it only tests what you tell it to. Though this may seem obvious, it’s important to keep in mind because even after unit, integration, and performance tests have passed, a single end-user can make the whole system crash in seconds. This usually happens if the user does something the developers did not expect.
This is why human testers are so valuable. They try to anticipate and mimic the behavior of end-users. Testers act as the users' ambassadors in the software development process. In other words, testers perform tests on the users' behalf.
Related reading: 5 Failures in Test Automation - and Best Practices for Tackling Them
Download the whitepaper to learn the basics of shift-left testing and how to find a simple and stable test automation solution that guarantees a great user experience.
When it comes to choosing how you will automate your tests, there are many different options. These fall under three main categories: code-based frameworks, low-code tools, and no-code tools.
Learn more about each of these with the following resources:
Based on the title of those resources, you can probably guess which type of automation we are advocating for. In the following, we will dig deeper into why we believe no-code automation is the way forward - even for teams with highly skilled developers on board.
The positive outcomes of test automation are clear. So why don’t all businesses automate their testing?
The answer can be found when taking a close look at most test automation tools.
So far, test automation tools have dictated that automating requires coding. Since test automation takes place within the IT department, you might argue that this doesn’t matter, because many people within IT are fully capable of programming.
There’s just one problem: the majority of testers aren’t programmers. They are business process experts. Code-based automation solutions prohibit them from contributing to their area of expertise: the quality of business processes.
So when test automation requires programming, it’s either pushed to the developers in the team, or it’s left with the tester to figure out and spend time on - either way, this skills gap ends up draining valuable resources and time, often creating bottlenecks, and costing more than could potentially be saved by replacing manual tasks with automation.
This is what we call the Test Automation Paradox - when setting up and maintaining test automation takes up more time and resources than what can be saved by automating.
This is what we call the Test Automation Paradox - when setting up and maintaining test automation takes up more time and resources than what can be saved by automating.
But it doesn’t have to be that way. By introducing automation that doesn’t require coding, teams can overcome the barriers that code-based solutions create, and see the benefits of automation, fast.
With no-code, testers don’t need knowledge of programming languages to design and build automated test cases. And with the barrier of developer dependency removed, they can match the pace of the release cycle, improve productivity, minimize risk, and the pressure on the team can thereby be removed.
Learn more about no-code test automation and the test automation paradox in our whitepaper: The No-code Way.
Whether it is the increasing alignment between IT and business, the rising adoption of agile and DevOps, or the focus on time to market, codeless automation is the enabler that the software industry has been waiting for.
In this whitepaper, we guide you through the basics of test automation, why it sometimes fails, and how to overcome these challenges.
No-code test automation is a game-changer for testers. A test automation tool that uses a visual language instead of code to describe tests is a much more efficient way for testers to work. Here are 7 reasons why.
Learn more about no-code test automation in our blog post How No-code Test Automation Closes the Skills Gap in Software Development.Why use no-code if you can code?
No-code empowers those without coding skills to create and contribute to test automation. But what about those who can already code? What do they stand to gain from adopting a no-code solution?
When you're building a test automation framework, it has to be well structured for it to take minimal work to maintain. It can take hundreds of valuable developer hours to build. This takes you away from value-creating tasks like developing new features. It also costs the business more than it should.
As a business grows, so too do the software requirements. New features are added, customizations are made. All of these changes require testing. But for code-based frameworks to keep up with the growing demand for testing, you need the resources to maintain the framework. The more it takes to maintain test automation, the less value it brings, and the more difficult it is to scale.
Taking over another developer's code isn't always worth it. Either, you dissect the code and find a way to make it work for you. Or, you scrap the project and start again. Both cases require a significant investment of time, which once again takes your time away from working on developing new features.
A common language closes the skills gap between developers and non-developers. It enables testers, business users, and developers to use the same approach to build and maintain test automation.
With a code-based solution, you have to comb through lines and lines of code every time a test breaks, or an update has been made to your system under test.
With a codeless solution, maintenance is simplified because the interface is easier to navigate.
Read the full article: Why Should I Use Scriptless Test Automation if I Can Code?
Agile, DevOps and CI/CD all go hand in hand and play a critical role in delivering quality at speed. Next is an overview of each of these methodologies, and a list of resources where you can learn more.
Agile is a methodology or approach to testing and development that aims to make the Software Delivery Life Cycle (SDLC) more flexible and effective. It’s made possible with tools such as test automation, which ensure continuous feedback from testing to development. Agile testing thereby makes businesses less vulnerable to changes in the market as they can adapt faster.
Get the Agile Testing whitepaper to learn how to work with the agile methodology in testing for successful software development.
DevOps is the bridging of development and operations. It’s mostly used as a job title, but it’s also an approach. DevOps has emerged in connection with the agile movement with the need to increase the frequency and speed of product releases while maintaining or improving a certain standard of quality. DevOps encourages collaboration and communication between teams from development to deployment, and requires automation throughout the pipeline in order to enable the flexibility and continuity that is required to deliver quality at speed today.
Get the DevOps and Test Automation whitepaper to learn how to automate testing in the DevOps lifecycle.
CI/CD stands for Continuous Integration and Continuous Delivery/Deployment - and that is exactly what is - a continuous, rather than a siloed, waterfall approach to software development. The CI/CD pipeline, which is typically built by the DevOps, is a series of automated tools that work together to secure a constant, accurate flow of code from development to deployment.
Watch the webinar on continuous testing to learn how you can achieve continuous delivery with automated testing.
This guide has introduced a new paradigm in test automation: No-code.
No-code test automation lowers the barriers to building and executing automation, for both technical experts and business experts, and helps teams avoid the common pitfalls in implementing automation.
Listen to our podcast episode of Decoding Automation to brush up on why no-code test automation helps teams close the skills gap in software development and build maintainable and scalable test automation.
Next, we'll take you through important steps in getting started with test automation. This includes finding the right test automation tool as well as following best practices and guidelines that will help you rethink or optimize your existing approach.
As a part of getting started with test automation, it's ideal to have a test automation strategy in place. Particularly when building test automation intended to be scaled at some point, it’s inadvisable to take an ad-hoc approach to automation.
Having a strategy in place does not counteract agility by putting rigid processes in place. Rather, it ensures that you take the most important factors for your specific business needs into consideration. This includes the size of your business, the industry it’s in, the technologies it uses, and the resources available.
We have created a test automation strategy checklist of factors to focus on which can be used to get started with test automation. It can also be used to optimize your current automation process, if you have already begun your automation journey.
The strategy checklist will walk you through everything from project scoping and choice of approach to execution plan and test case analysis.
Some of the items you might already be able to check off, while others will require some work - perhaps even help from external consultants. The strategy can be defined in broad terms as well as for more narrow projects.
When evaluating automation tools, go through the following checklist of requirements:
When laying the tracks for a test automation solution, it's important to consider what business value it brings.
You should start this process by asking three fundamental questions:
1. What challenge/challenges are you addressing with test automation?
2. What is the desired state once test automation has been implemented?
3. What is the desired outcome of the investment, and how can it be measured?
Being able to answer these questions and measure the value of test automation should be a core step in any investment you make. We've outlined more on this topic in our post on how to measure the business value of test automation.
To help you further, we've put together a brief article on what businesses miss when calculating the ROI of test automation.
From not factoring in implementation and maintenance costs, to underestimating the costs of incidents, we've outlined four common mistakes, and how to avoid them.
In addition to putting a strategy in place to achieve success with automation, there are also a number of best practices that should be followed to achieve the desired outcomes.
Most mistakes in test automation are predictable and can be avoided by following the best practices for building maintainable and scalable test automation.
Learn more about how to build maintainable and scalable test automation in our guide with best practices for building test automation.
There’s a major difference between automating twenty test cases and automating two thousand test cases; while it’s completely possible to take an ad-hoc approach when there are only a few test cases, it becomes an entirely different story when the numbers run into the thousands.
In this introductory guide, we outline best practices which you can follow as you build your suite of automated test cases.
Once you have an understanding of the best practices for building test automation, you'll need to decide how you want to create your test cases. There are essentially two ways to do this: with a test automation framework or with a codeless testing tool.
Which is the best solution for you and your team?
To answer that, let’s take a step back to talk about what a test (automation) case is, as seen from the perspective of the tester.
The elements of a test case:
All these elements are worth going into details about, but there’s one that is at the core of how we should approach test automation. It has to do with how to describe a step-by-step process.
A test case is essentially a description of a business process. For example, “log in to an application and check that the username is properly displayed,” or “put items in a shopping basket and check that the total including discount is correctly calculated”.
A test case is essentially a description of a business process.
If you ask a tester to illustrate one of their daily on-screen tasks, chances are that they will do it in one of two ways:
Drawing a step-by-step flowchart of user interface actions is an intuitive and flexible way to describe processes. Flowcharts are useful because they allow you to branch logic, add inputs from data sources, and much more.
For this reason, some test automation tools are developed entirely around the concept of visual GUI/UI flowcharts. For example, the Leapwork Automation Platform. In fact, there’s an entire industry standard for documenting business processes this way. It's called BPMN (Business Process Model and Notation).
Here’s an example: Testing a login form in a web application. It is simple to sketch out as a flowchart, where each “building block” consists of an action and an element in the application’s interface.
The following flowchart illustrates what an automated test case would look like. It's a regular login process consisting of the following steps:
The flowchart is not just a representation of an automated test case, it is a tool for actually activating and executing the test case. An automation tool built on this visual approach empowers testers to work with automation, without having to program. It enables them to create, maintain, and execute test automation in a much simpler way, by removing the unnecessary complexity of code.
There are several tools on the market with a visual approach to designing test cases, but many of them are based on programming disguised by a very superficial layer of visualization.
Below is an example of how a “visual automation tool" turns to programmer jargon as soon as something unexpected happens. In cases like these, the user must still understand programming terms and structures. For instance, when defining the parameters of an action or managing unexpected scenarios.
There's more to test automation than building test cases. Once you and your team have implemented a test automation tool and you've created a suite of automated tests that provide sufficient coverage for your quality assurance, you've reached the next stage in your automation journey.
There are a number of steps you can take to further improve your testing processes and increase the return on your investment in the test automation tool. Next, we'll take you through these steps.
Test automation lets you execute more tests, faster. This means you will also get more test results. So how can you handle those most effectively?
Setting your team up for success with test automation analysis is arguably just as important as achieving success with the implementation.
Here are four tips for analyzing test automation results:
Learn more about each of these steps in the factsheet: How to Analyze Test Automation Results
In extension of the best practices for analyzing test automation results outlined above, it is important to consider how to ensure environments that produce reliable test results. The quality of your test results is only as good as the quality of your test environments.
The main requirements for good test environments are:
1. Make the application in question testable by dealing with third-party system dependency. This can be done in one of two ways:
2. Make test environments fit your DevOps pipeline. This includes considerations related to load balancer, deployment, service accounts, environment installation, and more.
3. Decide how to manage test data. There are several ways to ensure that test data has a certain quality, including:
Learn more about how to set up high quality test environments in our blog post: Why Good Test Environments are Crucial for Successful Automation
Artificial Intelligence (AI) is an intriguing – and sometimes intimidating – phenomenon. It is no longer the stuff of a faraway future.
The world of software quality assurance dreams of artificial intelligence. The promise of a scalable, self-improving, digital workforce is too good to ignore. And in theory, there are endless ways in which AI can be utilized in test automation.
But, to date, no one has developed a computer program of general intelligence able to think on its own. The more advanced applications of AI in quality assurance, e.g. robots designing test cases on their own, are still fiction.
What does exist are clever statistical data analysis methods. We might call these artificial intelligence algorithms. Some of them are machine learning (ML) algorithms that can build clustering and predictive models out of even small amounts of data. Other algorithms mimic human decision-making, such as the way humans interact with software. Image recognition, for example, imitates humans' visual cognitive processing of what is on a computer screen. Algorithms like these are incredibly powerful when applied in the right way.
Learn more about AI, ML and hyperautomation:
Gartner define Robotic Process Automation (RPA) as follows:
"Robotic process automation (RPA) is a productivity tool that allows a user to configure one or more scripts (which some vendors refer to as “bots”) to activate specific keystrokes in an automated fashion. The result is that the bots can be used to mimic or emulate selected tasks (transaction steps) within an overall business or IT process. These may include manipulating data, passing data to and from different applications, triggering responses, or executing transactions. RPA uses a combination of user interface interaction and descriptor technologies. The scripts can overlay on one or more software applications."
As you may have noticed, RPA and test automation have several things in common: They are both about increasing productivity, they both emulate users to perform tasks, and they both utilize user interfaces to perform actions.
Most test automation tools are not suitable for RPA because they lack enterprise features. These include governance, change tracking, and audit trails. Interestingly, RPA projects are often kick-started with test automation tools! This is because they offer a pragmatic way to realize business-related automation potential.
Learn more about RPA: