Positions and Areas

Watch this video to learn how to use positions and areas.

What you will learn

  • How to get a Position.
  • How to Offset positions.
  • How to define an Area.
  • How to read data using Position and Areas

Download files: PositionsAndAreas

Lesson Overview

Watch this video to learn how to use positions and areas.

What you will learn

  • How to get a Position.
  • How to Offset positions.
  • How to define an Area.
  • How to read data using Position and Areas

Download files: PositionsAndAreas

In this video I will talk about Positions and Areas.

When working with text and image recognition this is an important thing to understand, in order to optimize the accuracy and the time it takes to find, click and select elements. I will occasionally shift to a presentation showing some definitions of positions and areas and then I’ll show how it’s actually used in Leaptest.

We will start by looking at Positions.

  • Open first slide in downloadable presentation

A position is basically a coordinate on a screen with the coordinate set (0,0) in the upper left corner.

  • Slide 2 in presentation

Moving from this location and into the screen will change the coordinates. The actual numbers depend on the resolution of the display, but is not really something you need to care about when working in Leaptest. You just need to understand that a position is a point on the screen.

  • Slide 3 in presentation

Normally you get positions from building blocks. For instance from a Find Image block that will search the screen for a specified image. Once the image is found, you can use the “Position found” property to get the position in the center of the image. You also have the possibility to adjust that position in the building block itself.

Another example is Find Text block, where the “Position found” will return the position of the upper left corner of the rectangle covering text found.

Let me just demonstrate this in Leaptest.

  • Focus on Leaptest.

I’ll add a Find Image block and just capture the image in the presentation.

  • Add Find Image
  • Capture Leaptest logo in Slide 3 in presentation

I will add a Hover Position block to show that the “position found” from the Find Image is actually in the middle of the image.

  • Add Hover position
  • Connect “position found” from Find Image to Hover Position

Let’s run the case

  • Run case

As we can see the mouse hovered directly over the center of the image.

  • Slide 4 in presentation

Sometimes you want to use the position from an image to set a new position. For instance, to define the area of a menu relative to a logo, we would have to offset the position found from the logo to where we want the area to start. We will look at areas a little later.

To move or offset a position we can use the Offset Position building block. I will show how to use this in the test case.

  • Focus in Leaptest

What I want is to hover the first image we find and then move the mouse to a different position. I will add an Offset Position block and use the image position as the starting point.

  • Add Offset Position block

To specify the vector by which the position should be offset I right-click the Offset value and click “Capture the position offset”.

  • Drag vector

I can now capture a vector by dragging the mouse. I will then add yet another Hover Position to move the mouse to the new position and wire up the building blocks.

  • Add Hover Position
  • Wire up blocks

Let’s run the case

  • Run case

As we can see, the mouse followed the vector to the new offset position. The next topic I’ll dig into is Areas which is heavily correlated to Positions.

  • Slide 5 in presentation

An area is a rectangle defined by 3 elements:
– The Position, which is the upper, left corner of the rectangle
– The width of the area and
– The Height of the area.

So in other words, the Position defines where the area is, and the width and the height decides the Size of the area.

  • Slide 6 in presentation

Just as with positions you can offset an area. In case of areas you can actually offset both the position and the size, or just one of them. I have illustrated this in the presentation.

When you offset the position, the area is just moved.

  • Slide 7 in presentation
  • Slide 8 in presentation

 

When you Offset the size, the size of the area is changed.

  • Slide 9 in presentation

To offset an area you can use the Offset Area building block, that allows you to change both position and size. I usually use another combination of building blocks when I want to move an area. Instead of using the Offset Area block, I use the combination of an Offset Position and then the Set Area blocks.

Let me demonstrate it

  • View demo application.

I have built a small table of data that I would like to iterate. In this example I will read the line number, get the position from that number, set a new position and add some areas over the Name and the Progress. In other words just read the name and the progress from the table for each row and print the result.

  • Focus on Leaptest

The first block I add is a Loop block.

  • Add Loop block

It will iterate from the minimum to the maximum value – in this case 1 to 5. For each number it will trigger the top connector. For each iteration I will find the number in the table. I will use a Find Text block to find the numbers.

  • Add Find Text

I will specify an area manually in which the numbers are expected to be found.

  • Specify area containing all row numbers

When the case runs, it will find the numbers in order, and I will use the position where the number is found, to specify the area where I can read the name in the row.

I will use an Offset position, to set the upper left corner of the area covering the name.

  • Add Offset position

I will draw an Offset vector from the upper-left corner of the found number to the upper left
corner of the table cell. This new position will be the upper-left corner of the area as well.

I add a Set Area block to specify the area.

  • Add Set Area

Finally I will add a Get Text block to read the text in the area specified.

  • Get Text.

I will connect the newly composed area as the input to the Get Text and run the case.

  • Run case

As we can see, we found the 5 row numbers and got the names returned from the Get Text.

The last thing to do is to Offset the area and read the progress as well. I will start from the area we defined to get the names, offset the position of this and use it to read the status.

  • Add Offset Area

I then add an offset value to the position just as before, by dragging a vector to the new position.

  • Drag vector to new position

Now the area has been moved to the next cell and I will use yet another Get Text to read the progress text.

Let’s run the case

  • Run case

As we can see in the log, it found row “5” and used the position of this to define an
area to find the name. Then offsetting this area to find the progress.

This ends this video showing you how to use positions and areas to optimize and
control image and text recognition.