In this video I will talk about the scheduling and reporting features in Leaptest.
Being able to schedule the run of test cases is an essential part of test automation and having an easy access to view the results is equally important.
When you want to create or change a schedule, you click on “Scheduling” in the main menu. This opens the list of existing schedules. In this case I have 2 schedules defined, but you can have as many as you want.
A schedule is basically a definition of how a controller should run a set of test cases on a set of environments. Environments are the machines where the agents are installed and you can have several of these environments defined. Once a schedule is created it is running on the controller. This means you define it in Leaptest Studio, but keeping track of the schedule and handling the execution on the environments is all done by the controller.
In the list of schedules you can see the title and the type, whether it’s enabled, the status of the schedule and some stats about the time for the last executed run and the next planned run. Further you can see the number of test cases and environments selected for the schedule.
If you right-click a schedule you have several options.
- Right-click Scheduled demo
On top of being able to create, edit and delete a schedule, you can force the schedule to run once, by selecting the “Run Now” option. This can be useful if you want to run the schedules manually or you just want to do a test run to see the schedule succeed. Once the schedule is running you can also force it to stop. This option is not active in this example as the schedule is not running. The last option is a toggle between enabling and disabling the schedule.
- Enable schedule
Once enabled you can see that the “Next run” is calculated on the controller and shown in the list.
- Disable schedule
The menu at the top contains the same possibilities as you have when you right-click a schedule. To create a new schedule I press “New” and the “Create New Schedule” form opens.
- Press [New]
I’m gonna start by adding a title and some description. By default the schedule is enabled and I’ll keep it like this.
- Tab ‘Schedule’
On the next tab I can select the “run type”. “Ad-hoc” means running it manually from the list of schedules or that it’s triggered from an external system through the Leaptest API. I’m not gonna cover the API in this video, but you can find material in the Learning center going through the API.
If I set the run type to Scheduled I can define when and how often the schedule should run. I can select the days of the week we want the schedule to run and what times a day it’s allowed to run. I can also specify how often it should be repeated and some settings that will keep the schedule from running wild in case of problems.
- Tab ‘Cases to run’
On the ‘Cases to run’ tab I select the test cases I would like to include in the schedule. I can only select test cases from the same project, so be aware of this when you decide how to organize your projects. I will select the “Open Leaptest.com” for this schedule. A test case can easily be in more than one schedule, but it’s definitely easier to administer if the test cases are in only one schedule. You can also create several schedules that together will run all test cases within one project. That’s all up to how you want to organize the schedules.
- Tab ‘Environments’
On the ‘Environments’ tab, you select the environments you want the test cases to be run on. The controller will distribute the selected test cases to the environment in parallel, so make sure that the selected test cases are intended for the selected environments. In this case I’ll select my Amazon environment – choosing “Local” would trigger the test cases to run on my local machine, which would have some impact on this video.
- Tab ‘Actions’
Part of setting up a schedule is to add some Actions. When you add an action you can select how the action should be triggered: It could be when the schedule is done or when a specified number of test cases in the schedule has failed.
- Add ‘Action’
Once the action trigger is set up, you can specify what to do once the action gets triggered. You can select between: sending a mail, doing a HTTP request or running a powershell script. All 3 are powerful tools to do notifications, integrations to 3rd party systemetc.
We have some videos and articles in the Learning Center describing integrations to for instance Jira, TFS and HPQC that is triggered from actions like this.
I’m selecting the E-mail option here, and click “show email details”. In here I can specify the recipient, the subject and the body. In the subject and body field I can right-click and select “insert token”. This gives access to some variables holding information about the schedule. For instance I could include the number of cases failed in the subject, and a list of links to the videos for the failing test cases in the body. All this will help the receiver of the mail to react fast and precise. You can also use the tokens when you set up the HTTP Request or Powershell actions.
I can add as many actions as I want and it’s easy to remove them again. I’m just gonna remove this for now.
- Remove action.
- Tab ‘History’
The last tab is a list of the historic runs of the schedule.
When I click Save, the schedule is added to the list of schedules and I can see that it has been queued and the time for the next run planned.
- Click [Save]
It should run in a minute or so, which bring us to the reporting section.
I click “Reporting” in the main menu. This brings up the list of reports – in this case only 1.
I’ll start by creating a new report.
- Click [New]
I’ll give it a title, and then I select the filtering conditions and how the test results should be grouped.
In this case I select the schedule as the filter and the date and the result as the groupings.
- Click [Save]
If needed you can export a report by clicking “Export” in the top menu.
- Click [Export]
In the File dialog select the format for the report and click save.
Once it’s saved it will open the file explorer so you can check the result straight away. Let’s just look at the PDF report.
- Open PDF report
Contains all test results, grouped and filtered.
- Open ‘Dashboards’ in main menu.
The last thing I’ll show in this video is the ‘Dashboard’. You can consider the dashboards a visual representation of the reports. So it’s an easy way to get an overview of the state of your schedules.
The Dashboard consists of one or more widgets that each can be configured to show data from only a project, a schedule and so forth. I’ll add a new widget to watch how our new schedule is doing.
- Click ‘Add widget’
When I click ‘Save’ it will take a few seconds for the controller to return all the data to the widget.
Here it is. So this pie chart tells us that all test cases have passed in a particular schedule today. If I want to I can right click the widget and turn it into a report.
- Right-click and click ‘View as report’
This is useful if you want to drill into details of why one or more test cases keep failing.
If you click ‘Save’ in the report view, the report will show up in the list of reports. Otherwise it will not be saved once you click away from the report.
This ends this video where I demonstrated the schedule and reporting features in Leaptest.