Building Automated Regression Tests: What to Include

Over time, as your application grows or changes, your regression test suite will grow as well, numbering into perhaps hundreds or thousands of regression test cases. This is why automation will inevitably become part of your testing process at some point.

Building an automated regression suite requires less effort than many think. The regression suite is essentially just a suite of existing test cases that you have already built that verify the functionality of the product. This can include everything from unit tests to integration tests.

Setting up an automated regression test

When you set up your automated tests, you may however want to do some spring cleaning to ensure that you only transfer those test cases that provide real insight into the quality of your product, and to remove tests that have become obsolete. This will make it easier for you to maintain your automated regression suite with time.

In this connection, ask yourself the following questions to identify test cases that should be included/excluded from your suite (if the answer is yes to the question, there’s a good chance that the test should be included):

• Does the test case check a business critical feature?
• Does the test case check a core product feature? 
• Does the test case check a feature that is highly visible to its users?
• Does the test case frequently yield bugs?
• Does the test case check a feature that has recently undergone changes?
• Does the test case check a new feature?
• Does the test case check an integration?

Remember that you can always adjust your testing suite. As your team develops and tests, you’ll be constantly learning and adapting, so it’s only natural that your test suite will too. When you start automating, you will be able to free up time spent on manual testing, letting testers focus on improving the quality of the tests.

It’s crucial, however, that the tool you use to set up your tests provides a good overview and can be adjusted by anyone on the QA team, as this will make it much easier to maintain, and hence improve tests over time.

How to find the balance between automated and manual regression testing

When setting up automated regression tests, the question of how many tests should be kept manual vs. automated typically arises.

Striking the perfect balance between manual and automated testing, first of all, requires an understanding of what automation can and cannot do.

How to do automated regression testing

Automation robots are designed to do exactly what you ask them to, nothing more, nothing less. While it might be easy to find a problem when you’re looking for it, it’s much more difficult to find a problem that you didn’t even know you should be looking for.

In other words, automated regression testing will help you find your known unknowns. It won’t help you find your unknown unknowns.

Only exploratory, critical, manual testing will enable you to find the unknown unknowns.

Testers will therefore continue to fill the critical task of monitoring, evaluating and updating the tests that they set up, as the software system changes or grows. It’s also their task to think outside the box and look for potential pitfalls in the system.

The great thing about automation is that it creates a positive cycle – the more tedious, repetitive tasks you automate, the more capacity you free up for you and your team, allowing you to find these pitfalls in existing functionality through exploratory testing.

Furthermore, it’s important to understand that it isn’t necessarily a matter of a test case being either 100% manual or 100% automated. A test case can be partly automated and should be if it includes highly repetitive tasks, such as logging in to an application or filling in user information. 

In conclusion, the ideal approach to regression testing includes a continuous focus on efficiency and time optimization through automation, as well as a critical mindset and careful evaluation of new and existing test cases.

By taking a balanced regression testing strategy approach with automation opportunities as a core focus, you can ensure that projects stay on track and on budget, that your team members are using their skills and capacity in the best way possible, and that, perhaps most importantly, your software stays bug-free, giving your end-users the best possible user experience.

Read more: What Automated Regression Testing Tool Works Best For Your Team

Read on in our next blog post on this topic: How frequently should you run your regression tests? or download our whitepaper: How to do Regression Testing in Agile Teams to learn everything you need to know about regression testing in agile teams.