The LEAP

Automation insights and productivity tips from LEAPWORK.

All Posts

How to Build Stable Test Automation

Test automation brings many benefits with it, such as increased execution and reduced risk. But these benefits may be diminished if the tests aren’t performing as intended. There are several reasons why tests become unstable, most of which you can turn around by following these best practices and guidelines.

Building test automation isn’t always smooth sailing. Even with the right tools and the right strategy, tests will fail unexpectedly from time to time. 

Following best practices will get you a long way, but if there are external factors that aren’t behaving as expected, there’s a good chance you’ll end up with unstable tests that fail left and right, even though the application under test is actually performing as it should.

Some of the factors that can impact the behavior of the application under test are:

  • Test environment performance
  • Hardware performance 
  • Network in which test is being executed 
  • Proxy configurations
  • Database bottlenecks 
  • Web server/test application slowness
  • Nightly group policy updates

The question then becomes how you can build tests that are stable, despite the presence of the above-listed challenges.

Here are a few tips and tricks that you can follow to solve some of these challenges, in the order that they are most likely to occur.

Timing

When building your flow, the first thing you want to think into the equation is timing.

Timing can make or break your test flows, but by following a few simple steps as you set up your tests, you can cross this bridge early and minimize the likelihood that timing will be the source of your problems.

Start by running your test locally and try to find the places where tests are failing due to timeouts. Increase timeouts within failing blocks first, as these help ensure stability in case your network or test environment is running slowly. If the test fails here, consider adjusting timeout, using await requests, or configuring the proxy.

Next, schedule the same test to be executed 5 to 10 times in a row on your agent and analyze the results. If your tests are still breaking, you should look beyond timing and reconsider the logic of the flow as well as any performance issues in the environment that could have occurred at the time of the test failure.

Once you’ve identified and fixed the above issues where possible, run your tests on schedule again. Ideally, you should keep running your tests, analyzing them, and fixing any errors within them, until you see no failed tests. 

If you don't reach this point within a reasonable time frame, go back to the list above and consider the external factors once again until you identify the error. Then, repeat your tests on schedule 10 times until all tests pass. At this point, you’re ready for production.

When running the tests 200 times, you should still expect to see a fail rate of about 1%, as it’s natural to see some failure that can be attributed to external conditions. 

Environment

Although errors may occur as a result of external conditions that are out of your scope, it doesn't mean that you cannot do anything to minimize the impact of these.

Here are a few ways to troubleshoot, as well as some best practices that you can follow to lower the risk of failing tests:

  • Have a dedicated environment for test automation so that it is not subject to overload. 
  • Prioritize your test environment highly, as you do your production environment, and give almost the same hardware power to it. 
  • Make sure you have a high bandwidth network with a firewall that doesn’t interfere with your test automation.
  • Make sure that the proxy is configured correctly.
  • Always keep the database administrator up to date with any database-related performance bottlenecks.

Last but not least, you should always - as this is after all the point of test automation - check that the reason for the failed test is not in fact the application under test.

These issues should be reported to the development team, in a continuous manner if possible, so that bugs can be fixed before anything is moved into production.

To learn more about best practices for test automation, read our guide to reducing risk, lowering costs, and driving value with test automation and sign up for our webinar on successful test automation strategy to learn how you can build stable, maintainable and scalable test automation that is both result-oriented and cost effective.

join test automation strategy webinar

Maria Homann
Maria Homann
Content Marketing Manager

Related Posts

How to Overcome Challenges in Cross-Platform Testing

With more businesses on the web, or moving to the web, it’s not often you see new desktop application releases. But the fact remains - many companies rely on a complicated web of desktop applications, web applications and legacy systems to perform business critical processes.

AI in Test Automation: 4 Types of AI that you’ll come across in 2021

AI has, once again, been mentioned as a key technology trend for the year ahead. But what does this mean for test automation? In this post, we break down key terms and explain how QA can benefit from these technologies in 2021 and onwards.

How to Automate Mobile Web Testing with Codeless Selenium

With brick and mortar stores closed during an on-going pandemic, websites, especially e-commerce websites, have to focus more than ever on creating quality customer experiences online. This has created a need for faster testing and new website functionality.