Differences between Web, Desktop UI, and Virtual Desktop Automation

Kasper Fehrend

Senior Product Evangelist at Leapwork

LEAPWORK supports advanced automation of desktop applications, web applications and applications running in a virtual desktop (Citrix, Remote Desktop, terminals, etc.). The different automation needs are enabled by different technologies. This article outlines the differences between the three automation types: Desktop UI, web, and virtual desktop automation.

Building blocks

All case types share most of the building blocks, but some building blocks are only available - and relevant - for some of the case types.

The Desktop UI case type supports all building blocks, and have one category of building blocks that is available only to Desktop UI cases: Desktop UI. This means you can use building blocks from both the Virtual Desktop and the Web Application case type in a Desktop UI case. E.g. you can start a browser as part of a desktop UI automation case using the web blocks or you can move the mouse to a position based on image recognition if needed.

The Virtual Desktop cases has two categories of building blocks that are are specialized for the Virtual Desktop cases: "Image and Text Recognition" and "Mouse And Keyboard". Both categories rely on image and text recognition (OCR) and can interact with everything happening on the screen which makes it a very good fit for virtual desktops!

Web Application is the case type with the fewest available building blocks. This is because of the flexibility needed when it comes to the execution of the automation flows (see later). The Web Application cases are obviously using the Web category for most operations.


The building blocks exclusive to the Virtual Desktop cases ("Mouse And Keyboard" and "Find And Get") are using Image & Text recognition and are hooking into the mouse and keyboard driver on the machine where the text case is executed, to drive the execution of the automation flow forward. In order to do so, the LEAPWORK Agent needs to be installed on the machine where the case is running. When an automation flow is executed, the LEAPWORK Agent continuously streams a video of what is happening on the screen back to the Controller. The LEAPWORK Agent inspects what is happening on the screen and uses this to drive the automation flow forward. So if you are executing Virtual Desktop cases these can only work on Environments of type "Remote (Agent)".

This is also true for the Desktop UI cases, which are also executed via the LEAPWORK Agent. In these cases, using the building blocks from the Desktop UI category, the LEAPWORK Agent is using Window UI Automation Framework to interact with both the operative system (Windows) and any desktop application based on Microsoft frameworks (MFE, .Net, WPF etc.). Any Java-based application is supported in a similar way. As for Virtual Desktop cases, the agent is streaming back the result and a video representation of the executing case to the Controller.

The Web Application blocks are using Selenium as the engine and they generate Selenium code on the fly when the test cases are executed. This means that a web case can be executed on all Environments that support/are based on Selenium, which includes the LEAPWORK Agent as well. If you are executing Web cases it can work on all types of the environment (currently: Remote, Selenium Grid, Sauce Labs, Browser Stack). 


When you create schedules you will typically start by selecting a number of test cases to include in the schedule. Depending on the selected automation flows you will only be allowed to select environments that fits all the selected flows.

For example, if you have selected one Desktop flow and three Web flows, the Desktop flow will limit the environments available to only the "Remote (Agent)" type.

Go to Learning Center.