Book demo Start trial

Capture text on screen

Watch this video to get an introduction to text recognition on virtual desktops, using the Get text building block.

What you learn in this lesson

  • How to find and mark text on the screen
  • How to use format and filtering
  • Iterating lists of text found on the screen

 

Lesson Overview

Watch this video to get an introduction to text recognition on virtual desktops, using the Get text building block.

What you learn in this lesson

  • How to find and mark text on the screen
  • How to use format and filtering
  • Iterating lists of text found on the screen

 

In this video I’ll introduce you to how you can work with text and numbers on the screen. It’s known as OCR – Object Character Recognition – but its also referred to as just ‘text recognition’.

We have a set of building blocks that covers this area:

  • Show menu

Find Text, Click Text and Select Text all search the screen – or an area on the screen –
for a specified piece of text. Then, depending on if the text is found, it clicks or selects the text
or simply just verifies that the text was found.

We also have a set of building blocks that should be used when you are looking for numbers on the screen. I will demonstrate this at the end of the video.

The “Get text” building block will extract whatever text is found in a specified area.

  • Add Get Text block

If no area is specified then it will search the entire screen which is rarely a good idea. You will get to much text back, and it will simply take too long, so when you think of text recognition, always consider how you can limit the area where the text should be found.

  • Show browser

I’m using the “Company” page on https://leaptest2.staging.wpengine.com as the example page for this demo. I’m starting by adding a Get Text block and defining an area on the web page that contains the text I’d like to capture. In this case the list of the address where Leaptest is present. Normally you would set the area relative to a logo or some other anchor point on the page, but I’ll just hard-select the area in this demonstration.

I’ll add a Pass block which is best practice.

Let’s run the case and see what is returned.

  • Run the case

The log will tell us what text was found. In this case we can see that it found all 3 addresses as one text.

I can add a format, so I will only get the text returned that matches the format specified. In this case I will add the word “Leaptest” before the “text” token. This means I will only get the text back that has the word “Leaptest” in front.

  • Add “Leaptest” in Format
  • Show browser
  • Run case

The log tells us that it found ‘San Francisco’. But what happened to the other 2 cities? If we look at the video, we can see that the case actually found 3 pieces of text matching the format. So when the building block finds multiple matching texts, it allows us to specify what occurrence to use or to iterate all of them. Let’s change the occurrence to 3 and see if the result changes.

  • Run the case

Voila. Copenhagen is now selected.

We can add a filter to further trim down the texts we are looking for. For instance we could add a “Contains” filter and simply just specify an “r”. This means it should only select the cities that contain an “r”.

  • Add “r” to Filter
  • Set “Set Occur” to “All”
  • Run the case.

In this case we can see that it has only found 2 cities matching the criteria we added.

When we set the “Use occur” to “All”, the building block will trigger the green output connector at the top for each text found. This means we basically trigger a workflow for each text found. To show this, I’ll add a Select Text building block to the Get Text building block. This building block will search for the text specified and select it with the mouse.

  • Add Select text
  • Set “Use occur” to ‘All’

I’m wiring the “text found” property to the Select Text, and also wiring the area to speed up the selection. Adding the area means that the building block only has to search a very small area for the text instead of the whole screen.

Now we should see “San Francisco” and “New York” selected one by one.

  • Run the case

So both texts were selected.

I just want to briefly mention two settings in the text blocks:

  • “Is Case Sensitive” – by default it’s not, so select this if you want it to be case sensitive.
  • “Is Dirty Edge mode” – if the text is very close to some graphics you can get a higher precision by enabling this setting. Don’t click it if you don’t need it, you’d better try to build the case without it.

The last thing I want to show is the Get Number building block. It’s essentially the same as Get Text, but it’s only looking for numbers. This means that it is optimized for numbers, so instead of using “Get text” and then convert it to a number, you can get a higher precision and performance using the get number instead.

Let me just demonstrate.

  • Delete Get Text
  • Delete Select Text
  • Add Get Number
  • Set area

Let’s run it.

  • Run the case

We can see that it returned the first number in the area, but also found the 2 other occurrences. Just as with the Get Text we could choose to iterate the numbers and use them in the workflow.

This ends this video focusing on text and number recognition.