.svg "Leapwork")
Knowledge base
Tips and tricks Best practice guides, FAQ & more
You’re probably here because you’ve decided to automate your regression tests. So far so good. But how will you get started? What should you consider? Should you just jump straight into it or take a strategic approach and prepare yourself for the project that lies ahead? Consider the 8 points outlined in this guide and you’ll probably know the answers to these questions by then.
Regression testing is a crucial part of testing as it helps reduce the risk of releasing with bugs.
But when it comes to introducing automation, there are many different approaches, and it can make it difficult to navigate the pool of options when it comes to defining the right approach for you and your QA team.
There isn’t necessarily one right approach to regression testing, but you can save your team a lot of time and resources by thinking the process through from the start. This will not only help increase output quality while containing costs.
How? By moving away from a reactive, put-out-fires-approach, and toward a strategic, plan-and-execute approach, you essentially decide to spend a few hours up front on deciding how you want to go about your regression testing, instead of spending weeks on programming later.
In short, having a regression testing strategy in place will save you lots of valuable time and resources during sprints, and instead let you focus on delivering quality at speed - something that is essential when testing in an agile team.
We’ve put together these eight steps to guide you through the thought process, and to hopefully save you some time down the road.
Make sure to also watch our on-demand webinar on automating regression testing to learn more about how you can make your regression testing more efficient.
Getting started with regression test automation strategy doesn't have to be difficult. Follow these eight steps to get off to a good start.
The first step to consider when introducing automation into your regression testing is the scope.
This includes outlining timelines and milestones for each sprint in the project, as well as defining which regression test cases should be automated and which should be kept manual.
Read more about how to find the right balance between automated and manual regression testing in: Building automated regression tests: What to include
It’s important that all team members (product owners, developers, testers, etc.) are on board with this scope, and that each know their responsibilities for the different parts of the project (we’ll go into further detail with this in the next step).
When choosing a regression test automation approach, there are three areas to consider: process, technology, and roles.
When building your automated regression testing suite, it’s important to have a well-defined and structured process. Make sure to cover the following in your plan:
Get the factsheet: How to analyze test results
Before you start, you’ll also need to identify which applications you need to automate, and what technologies these are based on. This will determine which test automation tool you should be using.
In most cases, regression testing will involve several application types: web-based, desktop-based, mobile apps, SAP/Dynamics/Oracle, etc., therefore it's important to have a tool that can handle all your automation requirements.
Many start out automating with a free, open-source tool such as Selenium, but then run into problems when they find out that it only covers some of their regression testing needs (Selenium only works with web-applications).
In addition, developers and testers often end up spending a substantial amount of time on writing automation scripts as well as maintaining all those scripts down the line.
Learn more about using Selenium for regression testing here.
Next, you need to define the roles for automation in your team. Regression testing may not be the only thing you’re automating, in which case keeping an overview of who does what is essential.
Examples of roles and responsibilities include:
Over time, your regression testing suite will inevitably grow as your software is developed, meaning more and more time will go towards maintenance of regression suites.
What’s more, it’ll become all the more important that there are clear roles defined for handling this suite (the more complex it gets, the higher the risk of breaking automation flows).
In this respect, it’s highly beneficial to use a regression testing tool that first of all lets you keep a clear overview of your testing suite. A visual, no-code regression testing tool, as opposed to one that requires coding, will make this a lot easier.
Secondly, it’s beneficial to have a tool that allows you to administer roles and access to automation flows and suites.
Learn more about how you can benefit from choosing a no-code automation tool for regression testing.
Risk analysis shouldn’t just be a part of your regression testing automation strategy, but your test automation strategy as a whole.
It can be a difficult and time-consuming task to try to foresee everything that can go wrong, to estimate the cost of this, and to find a way to mitigate or avoid that risk. Nonetheless, it’s important to at least consider these things.
Depending on your business’ size and the complexity and importance of your business processes, you can perform this risk analysis with the simple approach outlined in the example below, or you can move into a more extensive process of risk scenarios, cost calculations and mitigation planning.
The analysis below is created by listing the identifiable risks qualified with these details:
The next things to consider in your regression testing automation strategy are testing environments and data.
Organizations with a software department will typically have a more or less well-defined method for how software is released to production. This process usually includes one or more test environments.
Some release pipelines are mature and well-defined (i.e. a DevOps pipeline) and the work towards fast releases has either already begun or been deemed not relevant. In any case, it is important to evaluate the current state of your test environments.
Test automation is a “deterministic game”; known inputs will produce predictable outputs. This means that stable and predictable test environments are a prerequisite for successful test automation.
In relation to this, you should also consider the data that is part of the tests:
At this point you’ll already have considered the scope of your project in terms of timelines and responsibilities. This next step is about turning those elements into an actionable plan.
An execution plan should outline the day-to-day tasks and procedures related to your automated regression testing.
Before any automated test cases are added to the regression suite, it’s best practice to run and verify the tests multiple times to ensure that they run as expected. False failures are time-consuming, so it’s essential that test cases are robust and reliable.
It may be a good idea to create a procedure for making test cases resistant to changes in the system being automated.
This procedure will of course depend on the application in question, but it should cover how test cases recognize and interact with elements in the application under test. Execution of test cases should be handled either by the pipeline orchestrator (Jenkins, TFS, Bamboo, TeamCity, etc.) or by a scheduling tool.
This means that regression tests will run either as part of a build/deployment event or on a known time during the day. Also consider selecting a setup that allows for parallel execution of the test cases to get the feedback from the regression tests faster back to the development team.
Remember, you can never test too much, and the combination of test automation, reliable test cases, and scheduled/controlled execution will inevitably have a positive effect.
In any release pipeline, regardless of its complexity and maturity, there is a point at which a team needs to decide whether to release a build or not. Parts of this decision-making can be automated, while other parts still require human, critical thinking.
In other words, the final decision regarding release should ideally be based on a combination of algorithm results and manual inspection.
In any case, test automation results will play a critical role in this decision. But it’s up to you to decide if you only want to allow releases if all regression tests pass, or if you want to have the lead tester approve the result.
After a complete run of regression tests, you should consider including the application logs as part of the release decision. If the regression tests have good application coverage, then any errors not related to the UI should reveal themselves in the log files.
Having a plan for how to analyze failing test cases and the actions required to take afterwards is a critical—yet sometimes neglected—part of test automation strategy.
The time it takes from the point where a tester is notified of a failing test case until the fix is described, understood, and accepted in the development backlog is usually much longer than teams anticipate. As a consequence, release cycles risk being delayed, and the agile team becomes less agile.
Having a well-defined process for this can save a lot of time and frustration throughout the release cycle. A good place to start is to outline how different errors should be handled and by whom. For example:
Finally, once you’ve drafted your regression testing automation strategy, you’ll want to make sure that it is reviewed and approved by all members of the involved development team.
From here, try to work on enforcing a culture for continuous learning and improvement, where you include and embrace feedback from stakeholders, peers, and all team members working with automation and adjust the strategy where needed.
What's next?
Once you have a relatively good idea of how you want to approach your automated regression testing, and you’ve made an effort to plan the automation journey that lies ahead of you, the next step might be to find the right regression testing automation tool.
Watch our on-demand webinar on regression testing to learn more about getting started with automated regression testing, and to see Leapwork’s no-code automated regression testing tool in action.
