Skip to content

How to Do a Test Automation Tool Evaluation

Aske Denning

Aske Denning

Steep learning curves make it difficult to evaluate automation tools within a reasonable time frame. This can cause the evaluation to drag out and in the worst cases, automation projects are tanked altogether.

This post outlines some simple steps for evaluating and implementing automation tools. It is based on our experience that it is possible to get started with test automation in a matter of weeks, not months.

1. Define evaluation criteria

When choosing an automation tool, whether it is for automating tests or business processes, it is important to set up some criteria for evaluating whether the tool is the right fit for your organization.

For the right automation tool there needs to be a fit in the following three areas:

  • Technology fit: Does the tool support our systems and applications?
  • Process fit: How easy is it to build automation flows with this tool?
  • Organizational fit: Is the tool vendor able to provide necessary documentation?

The evaluation phase should not have to take more than five days. If it takes much longer to do an immediate evaluation of a tool, it might be because the tool brings with it an unnecessary amount of complexity.

Technology fit

Define the technology that you are looking to automate, and then find out if the automation tool in question supports that technology. In most cases, you can place it in one of the following main categories:

  • Web-based technology or mobile web application

  • Desktop-based application

  • SAP-based application

  • Citrix or other virtual desktop-based technology

Process fit

Start by outlining two or three flows that you want to automate. We recommend identifying some common processes that are often repeated and which, if automated, would bring about significant efficiency gains.Then use these cases for evaluating a tool’s user-friendliness; how easy is it to build these flows as automation cases with the respective tool?

Organizational fit

To make sure that an automation tool is a good fit for your organization, you should distribute the relevant documentation and materials related to the tool to the respective stakeholders:

  • IT
  • Operations
  • DevOps
  • Security
  • Management and business owners
  • etc.

2. Implementation

Once the alternatives have been evaluated, and you’ve chosen the tool to proceed with, it’s time to roll it out in your organization. Implementation consists of two main activities: Training and deployment.

Training

Ideally, the automation tool vendor should provide both a training program and readily available training materials designed to facilitate an advanced and fast internal adoption of the tool. The goal of a training program should be to enable participants to implement complete automation within their business domain.

DOWNLOAD SAMPLE: Automation training program

Example of a training program:

Day 1 – Onsite:

  • Introduction to the tool
  • Case-based learning with hands-on exercises
  • Self-study: The users presents up to five cases to work with during training week

Days 2-4 – Online:

  • Daily progress follow-ups
  • Online sessions with automation specialists as needed
  • Ongoing tracking of key challenges

Day 5 – Onsite:

  • Introduction to the tool’s advanced features
  • Status on key challenges
  • Presentation of solutions created during self-study

Deployment

The purpose of this phase is to ensure that the components of the automation tool are installed properly in a “best fit” deployment from the get-go and that scheduled runs of automation flows work as intended.

The deployment phase consists of the following deliverables:

  • Assessment and definition of installation and deployment method
  • Establishing required servers and environments
  • “Ready to launch” installation and configuration
  • Instructions on management and maintenance of deployed platform

Download the checklist and start evaluating tools with more confidence while staying organized in the process.

New call-to-action