Book demo Start trial

Automation using image recognition

Watch this video to learn how to use image recognition in your flows.

What you learn in this lesson

  • How to find images on the screen.
  • How to capture and edit images.
  • How to branch your flow based on image recognition.
  • How to use scroll to find images.

Go to next lesson.

Lesson Overview

Watch this video to learn how to use image recognition in your flows.

What you learn in this lesson

  • How to find images on the screen.
  • How to capture and edit images.
  • How to branch your flow based on image recognition.
  • How to use scroll to find images.

Go to next lesson.

In this video I’m gonna show you how to find and capture images when you create test cases in Leaptest.

First I’m gonna add the “Find Image” building block to my workflow. Using type ahead it’s easy to find.

  • Add Find Image building block.

The Find Image building block will search the screen for an image that you specify when the test case is designed. The Click image building block is almost identical, so the things demonstrated in this video will also go for the click image building block.

I have a browser opened in the background and it’s showing the product page on the US version of lego.com. I’m gonna use this browser to demonstrate some of the features regarding finding images. In a normal test case the opening of the browser and the navigation to Lego.com would of course be included in the test case itself.

The product page contains a nice set of images, and I’ll start by trying to find the Star Wars product image. I’ll Click “Select” and “Capture” in the building block and select the Star Wars image.

  • Click ‘Select images to find’ – ‘Capture new image’

Let’s run the test case.

  • Run the test case
  • Click the log entry “Found image”

As you can see the image was found.

  • Enlarge video section

Looking closely at the video you will be able to see a green rectangle telling you where the image was found. This is interesting for debug and verification purposes.

  • Enable edit mode

I’ll enable Edit Mode by clicking the orange pen.

  • ‘Expand’ Find Image

As you can see the building block has a green output connector named “Not found”. This connector is triggered if the test case hasn’t been able to find the image within the time specified in the “Timeout” property. If the image is found the normal green connector at the top will be triggered.

I’ll add some building blocks to illustrate how to use the 2 different connectors. I’ll add a Fail building block to the “Not found” connector and a Pass building block to the top connector.

  • Add Fail and Pass to the connectors

Let’s run the test case again

  • Run the test case

As you can see, the image was found and the test case ended by highlighting the Pass building block. In case the image was not on the screen, the test case would have ended in the status Failed.

The Find Image building block is typically used for 3 purposes.

  1. When you are loading a web page or you are opening a windows application, you will typically use a Find Image building block to tell you when the web page of the application is opened.
    For instance, I could have used the Find Image building block with the LEGO logo to tell me when the page was loaded. The connector at the top won’t trigger until the image is found, so it’s an implicit wait-operation.
  2. Find image is very useful when you want to have a known position on a web page or in an application. For instance when you search for text using OCR, it’s good to minimize the area where the test case has to search for the text. In order to make the test case more robust the position of this area can be set relative to the position found with the Find Image building block. You can set the exact position on the image by clicking the image in the building block and moving the cross hair.
    • Show the cross hair
  3. Probably the most used purpose is to do the final assessment. “Did this image appear when I clicked the button?” and then add the appropriate Pass and Fail building blocks.

Another interesting property on the Find Image building block is the “Await no movement” property. This is typically used when dealing with web pages, but can just as well be used when dealing with windows applications.

Especially modern web pages have a tendency to load some visual elements to the screen, and then reshuffle them into place after a short while.
The “Find image” building block is analyzing what’s happening on the screen, maybe every 10th millisecond, so it can easily happen to find an image in the first position where it’s shown on the screen. But this position will not be the right one after a short while, which could make the test case fail for the wrong reasons. In order to handle this, you can enable the “Await no movement” property.

  • Enable the “Await no movement”

This will instruct the “Find image” building block, to wait until no change has happened to the screen, for the amount of seconds specified in the corresponding “Timeout” property. The default is 3 seconds, but 1 second is sufficient for most cases.

  • Disable the “Await no movement”

Be aware that if the web page has a video running somewhere this property might not be the best idea. Instead you can add a Wait building block and specify the amount of seconds to wait until the “Find image” should search for the image.

  • Move elements
  • Insert Wait block

In this case we will wait 3 seconds from the start of the test case before the Find Image starts searching for the image.

Another helpful property is the “Await not found”. This is somehow the inverted operation of the “find image” operation. If this property is enabled then the “Find image” building block will not trigger the green output connector at the top, as long as the image can still be found. This can be useful if you are testing an info box or if an image is to fade away after a period of time.

Sometimes the image you are looking for is below the fold – meaning you need to scroll down to actually find the image. Leaptest is using image recognition in order to find the image, the test case has to scroll down to get the image into the visible area of the screen. The scroll feature can be used in a lot of building blocks so it’s important to understand the possibilities of this.

I’m just gonna remove the Wait block and replace it by a “Click position” block.

  • Remove Wait block
  • Add Click Position block

I’m using this building block to ensure that we are scrolling in the right window – in this case the list of LEGO products. I click somewhere on the page so the focus is set on the page before the scroll event is sent to the browser. Otherwise the scroll could happen in another window or simply not be executed.

  • Capture position in browser

I will then capture an image somewhere at the bottom of the product page.

  • Scroll in the browser.

I’ll use the “Chima” image.

  • Scroll back to the top.
  • Set focus on Leaptest

Then I’m setting the “Scroll to find” property to “Down”. This tells the building block to start scrolling down if the image is not found. It’s important to understand that the building block will send a scroll instruction to the element that has focus. This is why I start the test case with “Click image” building block, that clicks inside the web page. So if you want to scroll for instance a menu, make sure that the menu has focus by clicking somewhere inside it before scrolling.

I know that we will have to scroll so I’ll adjust the Timeout to 1 second. No need to wait 10 seconds before we start scrolling.

The “Max repeats” specifies how many scroll attempts should be performed before the building block fails.

The “Amount” property specifies how much to scroll on each scroll event. Each amount is the same as a click on the mouse wheel, so 3 is typically around one fourth of a screen height.

The “Delay” property tells the building block how much time to wait between the scroll events.

Let’s run the test case to see the scrolling in action.

  • Run test case

As you can see the image was found after a number of scrolls.

The last thing I’m gonna talk about in this video is the precision properties you can set on the image blocks. I’m not going into details with image recognition in general in this video, but will just introduce the 2 properties under precision.

  • Open Precision category on Find Image

If the resolution on the machine, where the test case runs, is different from when the image was captured, it could mean that the pixels are not exactly the same. It could also have an impact on the color density of the image. This means the image recognition will not be able to find a perfect match and in cases like this it can be necessary to lower the accepted precision for the pixel and color match. If you find yourself in this situation just lower it one step and re-run the test case until you find a level where the image can be found and the test case is still robust.

This ends this video giving a basic introduction to working with images and image recognition in Leaptest.