Skip to content

Test Automation vs. Manual Testing: 10 Considerations

Kasper Fehrend

Kasper Fehrend

Once a QA team has made the case for automation and is moving from doing only manual testing to introducing test automation, the change of process can come with challenges that need to be addressed.

A few examples of common challenges related to automation roll-out are:

  • If the automation of test cases is owned by only a few people, and not everyone on the test team, there is a risk that the ownership of product quality falls between the cracks.
  • If a DevOps team sends out new, “one-size fits all” directions and protocols for software releases, it might result in development and test teams losing equilibrium: They are still accountable for product quality, but they no longer have the power to do something about it.

To increase the chances of success when equipping a QA team with a test automation tool, it’s important to realize that manual testing and test automation complement each other, and that both have strengths and weaknesses. Below are ten aspects to consider when figuring out how your QA team can benefit from test automation.

Access guide to moving from manual to automation testing

1. Flexibility in testing

Manual testing comes with the inherent benefit of offering testers full control and flexibility of how to execute single tests. Testing manually allows fast, ad hoc testing and is crucial for signing off new features during a sprint.

Test automation should only be applied to more stable features. This means that automation cases for features are built one or two sprints after the implementation of the features in question.

2. Regression testing

Manual regression testing is incredibly time-consuming and comes with a range of issues:

  • Uncertainty about whether tests are executed the same way every time.
  • Executing the full regression suite manually creates a bottleneck in the release cycle making it rigid.
  • Due to time constraints, manual regression testing is usually not feasible every time the software changes.

For these reasons and more, regression testing is an ideal target for automation. Automated regression testing comes with following benefits:

  • Robots never sleep. This significantly extends the time window for running the regression suite.
  • Automated cases are executed the same way every time producing credible test results.
  • The time spent building an automated test case is a one-time cost.
  • An automated test case can be reused as many times as needed.

test automation vs manual testing3. Scalability

Manual testing can only be scaled with more people and hours being allocated to the given project. More test cases require more hands, which doesn’t scale well.

Automation enables a high level of scalability as you can simply just add more test executors (i.e. “robots” or “agents”).

4. The human touch in DevOps

DevOps is all about streamlining the release pipeline, and automation plays a key role in this. Test automation, specifically, is completely in line with DevOps principles, for various reasons:

  • Automated execution
  • Tests are triggered as part of a release pipeline
  • The development team is provided with feedback quickly
  • Reporting is highly structured

However, there are elements of the DevOps pipeline that require a human touch:

  • As mentioned earlier, testing of new features in a sprint manually comes with some advantages.
  • Analyzing and managing reasons for why automated test cases fail is still very much a task that requires a human mind.
  • Building automation cases is a manual task.

Read more: How to Automate Functional UI Tests in a DevOps World

5. Processes and ownership

When you apply automation to the DevOps pipeline, make sure that testers still have – and feel they have – a say in how the pipeline is defined. Testers are accountable for product quality and they should decide how often builds are pushed to test environments, when automated test should run, and how test results are distributed and formatted.

A DevOps pipeline should support the software development process, not the other way around. In the same way, a test automation platform should support, not dictate, the defined pipeline.

software delivery processTest automation supports the DevOps pipeline which in turn supports the software development process.

A test automation platform should come with plugins for pipeline orchestrators (Jenkins, Bamboo, TFS, etc.) and/or with APIs for executing test cases and to read test results.

Not two organizations or products are the same, so it’s important that a test automation platform has the necessary flexibility to support many types of software development processes.

6. Change management

Introducing agile processes and DevOps is a big change for organizations, and management must equip their teams to being able to handle this transformation.

Testers should have a chance to take a step back and reflect on how things are done today and whether processes can be improved. In a QA team both developers and testers should have a say, and each team member should be empowered to make the necessary changes.

Some test automation platforms are restrictive in the sense that they force testers into one way of working. To avoid this be aware that the testers, not the tool, should decide on the processes related to test automation. This helps ensure ownership in a change process.

Read more: How to Achieve Agile Testing with Test Automation

7. Tooling

Introducing an automation tool comes with the risk that a long learning curve or technical challenges take up too much time and require a lot of support between team members. This takes away time and focus from testers’ primary tasks (building test cases, analyzing requirements, reporting, etc.).

When choosing an automation platform, make sure that the evaluation is done thoroughly. Every member of a QA team should be able to work with it, and the tool should be a good fit in three aspects: technology, processes, and organization.

From a technical perspective, you can use the following requirements for the evaluation:

  • Which applications are you automating? (Web, Desktop, Citrix, other?)
  • Should the automation platform be deployed on-premises or in the cloud, or a combination?
  • Should test cases run locally, on cloud-based services, or both?
  • How well does the platform integrate with the release pipeline?

test automation tool fit

Criteria for evaluating a test automation tool: Organizational fit, technology fit, and process fit.

8. Skills

The essence of testing is to guard and improve the quality of an application through a series of tasks: Test case definition, requirement analysis, test execution, regression testing, result reporting etc. All these disciplines are still relevant and required when test automation is introduced.

However, it is crucial to ensure that the team can reap the benefits of test automation. Otherwise it’s just another set of tasks added on top of the already existing tasks. Upgrade the skills of the test team through training and make sure to select an automation platform that matches the skills of the team.

When it comes to the skill sets required for test automation, there are basically two different paradigms. One is programming-based requiring testers to code when building automation cases. Automation tools of this paradigm are designed with a developer mindset and assume that testers have a deep understanding of how the underlying technicalities of the system under test.

The other approach is to let testers build automation cases by working with a visual designer. With this method, testers can fully utilize their domain knowledge (see below), without having to worry about an application’s underlying code.

Read more: A Leapwork guide to reducing risk, lowering costs, and driving value with test automation.

9. Domain knowledge

A deep understanding of an application never goes out of fashion! As mentioned above, the concept of knowing a software well is often confused with having to know the code the software is made of. To be clear:

Programming skills are not, and should not be, a default part of a tester’s skill set, but domain knowledge will always be part of the testing profession – automation or not.

Utilizing automation is based on the tester’s knowledge about an application and how it should be tested. Make sure to nurse and value domain knowledge as this is the basis for good software quality.

Automation success is dependent on the collective knowledge of the entire test team. A robot is not a sentient being – it does what humans instruct it to do. So, implementing automation is about utilizing a team’s collective knowledge in the development of automation cases and making sure that automation becomes part of the daily work routine – in the same way that writing test scripts is a natural part of manual testing.

Read more: Will AI Take Over Test Automation?

10. Organization

Test teams are often organized around products or projects for changing products.

With or without automation, this a valid and efficient way of organizing teams. It results in dedicated, relatively static teams with a well-kept pool of domain knowledge, which is exactly what is needed for creating and maintaining a regression suite for an application. Make sure to preserve all processes in which testers are picking up additional domain knowledge about the application in question.

Success with automation requires that the entire test team is on-boarded and receives proper training. Otherwise, the overall responsibility for the product quality is neglected.

All members of a test team play an important role in automation, but not necessarily the same role. For example, testers with some technical proficiency can take on organizing reusable components and working with DevOps to include test automation in the release pipeline. Testers with a high level of domain knowledge can oversee creating and maintaining test cases and monitoring results of automated tests.

roles in test automationAll members of a test team, incl. domain specialists, technical testers, and test management play an important role in making test automation work.

Conclusion: The 10 considerations in manual testing vs. automated testing

 

Manual testing considerations

Test automation considerations

1. Flexibility in testing

Manual testing allows for fast sign-off of features on an ad hoc basis. But flexibility decreases as the workload increases.

Gradually building automated cases on a sprint-by-sprint basis provides a high degree of flexibility.

2. Regression testing

Manual regression testing is time-consuming, causes uncertainty and creates bottlenecks. Inevitably the regression suite grows to a point at which it can’t be managed manually.

Automated regression tests can be executed 24/7, the same way every time. Building automation cases is a one-time effort, and once built, they can be reused indefinitely.

3. Scalability

Scaling manual testing requires more people and more hours.

Scaling automated testing is done by simply adding more test executioners (robots or agents).

4. The DevOps pipeline

Several aspects of the DevOps pipeline still require human cognitive skills.

Test automation perfectly supports DevOps principles of streamlining the pipeline.

5. Processes and ownership

Testers should have a say in defining the pipeline, esp. the areas related to test automation.

A test automation platform should support—not dictate—your DevOps pipeline.

6. Change management

In a QA team, both developers and testers should be empowered to make the changes required to improve the software development process.

Testers—not the automation tool—should decide on the processes related to test automation.

7. Tooling

Tools with long learning curves and unnecessary complexity will require that a lot of resources are spent on support.

An automation platform should be a good fit in three aspects: technology, processes, and organization.

8. Skills

The most important skill of the tester profession is to utilize one’s domain knowledge of the system under test when defining test cases, analyzing requirements, reporting results, etc.

Programming skills are not—and should not be—a default part of a tester’s skill set.

Some test automation tools require testers to code when building automation cases. Others let testers build automation cases by working with a visual designer. With the latter method, testers can fully utilize their domain knowledge, without having to worry about an application’s underlying code.

9. Domain knowledge

Domain knowledge will always be part of the testing profession – automation or not.

Implementing automation is about utilizing a team’s collective knowledge in the development of automation cases and making sure that automation becomes part of the daily work routine.

10. Organization

Organizing test teams around products results in dedicated, relatively static teams with a well-maintained pool of domain knowledge.

All members of a test team play an important role in automation – but not necessarily the same role. Tasks will vary depending on technical proficiency and level of domain knowledge.

Get the Guide to Moving from Manual to Automated Testing