Testing in SAP: An Introduction to Automation Approaches

The difference between configuration and customization in SAP

• Configuration. Using a native tool to change a feature or behavior.
• Customization. A modification, feature, or extension is created through a special implementation or custom coding.

Examples of common integration tests in SAP

• TPP order system integrations. Making sure that new purchase orders are immediately made available and transmitted to SAP.
• CRM integrations. Companies need to sync data between their ERP (e.g., SAP) and CRM (e.g., Salesforce) systems. If, for example, a new customer is onboarded, the CRM data must sync with financial modules, performance management and potentially other business functions within SAP.
• Supplier system integrations. If a purchase requisition is created for a raw material or component in SAP, once the purchase has been approved it needs to be sent to suppliers. These suppliers are inevitably using separate systems when they respond to the request with a quote. These quotes are routed back to SAP for processing. To make the process more efficient, SAP would likely integrate with the supplier systems.

As we established in the previous section, upgrades, customizations, and integrations mean that tests must be carried out in SAP on an ongoing basis.

There are several types of tests that SAP testers must perform:

• Unit testing. In short, unit testing is testing a specific functionality in silo. It would take place when a WRICEF is complete, or a configuration is complete. For example, if you are working on a form, you would carry out a unit test on each item of the form. Unit testing makes sure that the configurations or customizations are working as they should. It is typically done once a “unit” is complete. The developer who created the configuration will often be the one to test it, otherwise, it is done by a consultant hired for the purpose. This takes place in a development system.
• Integration testing. Once all units are complete, you would typically move on to integration testing. In this stage, you test all the different pieces together. It’s an end-to-end test walking through business processes from A to B to make sure they work. In this stage, you could be testing an end-to-end process across different SAP modules, and across non-SAP systems. This is typically carried out in more than one cycle by a dedicated test team, a consultant, and a couple of key business users. Unlike unit testing, integration testing is carried out in a quality system and test system (in some cases). To make integration testing more efficient, this stage can be automated.
• Performance testing. Performance testing makes sure that functional parts of the system are responsive when in use. This could happen on an input form or document on SAP. Performance testing is also called stress testing and it is often automated. It can be done in conjunction with integration testing, but it can also be done after integration testing. It is normally carried out by an IT team, or a dedicated performance testing system.
• Functional testing. Functional testing consists of reviewing design documents and creating test artifacts including test requirements, test scenarios, and test cases. Functional testing is usually done by a testing team with a background in the SAP module being tested. Many businesses choose to automate functional testing to speed up the testing process.
• User acceptance testing. At this stage, the system will be in a mature state, most bugs will be resolved, and end-users will be testing. Testers either follow a test script, or they perform exploratory testing. Like integration testing, it will also take place in a quality system, and it is one of the last phases before going live.
• Regression testing. Once everything is in production and users are active, an error might be found. Or a user could request a new feature. Here, testers make sure that everything in production, and any new features that are released after the live date, work. Regression testing makes sure that whenever a change is made, the whole system is tested to verify that the change has not caused issues in another part of the system. It can be done manually, however, most businesses automate regression testing.

For the remainder of this article, we will be referring to migration, integration, and regression testing most frequently. It is within these stages that most development projects are held up. There are many reasons for this.

However, two challenges we see occurring frequently are related to manual testing (time constraints), and automation testing (specifically the problems that come with scripting and maintaining automation). The problem with manual and automated testing in SAP will be explained in more detail below.

SAP is a vast system and can cover most major and minor processes in a business, from invoicing to recruitment, and supply chain management. Even the tiniest error can cause wide-scale failure across the system. Naturally, testing has become a fundamental part of the development process.

With configuration and customizations taking place frequently, SAP and its integrations must be regularly regression tested to verify that the system is not negatively impacted by a change.

To do this process manually is incredibly challenging. But for many, it is the only option.

Not all enterprises have available development resources to build automated tests. Naturally, the responsibility is divided between those who are experts in the system – business users.

In the likely situation that you are assigned manual tests, it can take several days. All the while, business-essential work is being ignored. This slows down the release of customizations and configurations to a live environment.

In sum, when testing is performed manually, it takes longer, and there is a higher risk of error. For enterprises with large and complex digital ecosystems, relying completely on manual testing is not an option.

Below you’ll find an overview of the automated testing solutions, from code-based solutions to low-code and codeless solutions.

Before starting this section, it is important to note that it is a challenge for SAP to implement and agree on the best practices and standards for SAP test automation.

There are solutions available that can make SAP testing faster, easier, and with less maintenance. But a one-size-fits-all solution for testing does not exist. Every SAP environment and organizational structure will be unique, hence the right solution for one business may not be the same for another.

Testing SAP with the help of automation is the natural solution to make the development process more efficient. It doesn’t require days of manual testing, and regression testing can be run with the click of a button.

Related reading: How Can SAP Be Automated?

The downside is that many automation vendors offer highly technical solutions that require expensive developer resources to use and maintain. Because SAP is built upon a proprietary programming language (SAP ABAP), it is harder to find a suitable tool outside of SAP’s own offerings (e.g., SAP TAO and eCATT).

With SAP TAO coming to an end, this limits the selection further.

Below you can find a summary of the most common (code-based) solutions used by businesses today, along with their advantages and disadvantages.

SAP automation testing using Selenium

Selenium is a popular testing tool used by testers with technical skills to automate testing processes like functional testing and regression testing.

It can be used across browsers and operating systems, and it works with several programming languages making it ideal for companies with complex software spanning across web browsers.

The benefits of Selenium for SAP

• It’s free, as opposed to most other automation tools that include licensing costs
• It’s open-source, allowing extension and modification of source code
• It supports multiple programming languages
• It supports most operating systems
• It supports all major browsers
• It has community support, due to the large network of users
• It has integration options, allowing e.g., parallel testing and reporting

The drawbacks of Selenium for SAP

• It’s difficult to set up and use, potentially leading to high start-up costs
• It’s difficult to maintain
• Test development is slow as all test cases need to be scripted
• It only supports web browsers, not mobile, desktop, virtual or desktop
• Integration with the rest of the pipeline is difficult
• There is no dedicated support, only user communities
• It requires third party solutions to cover all testing needs

Extended Computer Aided Test Tool (eCATT)

eCatt is SAP’s in-built functional testing and integration testing tool and offers an ABAP script editor. Users create and execute automated tests of business processes in SAP. A common use for eCATT is automating regression testing if the system is extended or a support package is installed.

The benefits of eCATT

• It’s free of charge so you don’t need to acquire extra licenses
• It’s managed and maintained in one central system
• You can access all layers of an SAP system (database, application layer, and presentation layer)
• A modular approach to testing means users can reuse test and system data
• It can integrate external tools for extending reach to non-SAP systems

The drawbacks of eCATT

• Limited to SAP UI and API technologies. This means it only works in an ABAP development environment. HTML, for example, is not supported.
• Non-technical testers and business users can’t use this tool as you must know how to program
• The upkeep of the test script requires a lot of maintenance work

Quick Test Professional (QTP)

QTP is a keyword-driven automation tool for functional testing in web and desktop applications. It is based on the VB scripting language. Organizations often chose to go with QTP rather than eCATT as it supports technologies other than SAP GUI.

The benefits of QTP

• Record and playback features for quick test case design
• Comprehensive training material available
• Built-in REST API

The drawbacks of QTP

• Without a structured approach, the maintenance of QTP scripts can render automation code unusable
• Script-based testing prevents non-technical testers from automating complex tests
• Adoption time can take between 6-9 months
• Programming is required for cross-technology support

SAP TAO (Test Acceleration and Optimization)

SAP TAO is used as an aid for building automated tests in SAP. It is primarily used for end-to-end tests because it helps to break down software into smaller components, making automation more manageable. However, the tool is being phased out, so businesses using TAO will need to search for an alternative.

You can learn about the benefits and drawbacks of SAP TAO in this comparison chart, where we compare different testing solutions for SAP.

Restrictions of SAP’s own testing tools

As we established in the previous section, SAP’s own testing tools are limited in terms of what you can test. For example, if you are using eCATT, you can access all layers of an SAP system, but you will not be able to automate beyond SAP UI and API technologies.

Scripting automation

Most tools for test automation offer a function where you can freestyle programming. Using this function does not always make sense. It can increase the maintenance work required to keep the automation running, and it can make it harder to reuse tests.

For example, if incorrect data types are used and naming conventions for scripts and parameters are not followed, it will be difficult for others to reuse the script.

Reusing test automation script

As those working in big organizations will know, trying to get people to reuse the same code is a never-ending struggle.

New people come on board and will have their own preferences for scripting. Or a user may struggle to find reusable modules. Rather than spending the time identifying the reusable script, it can seem easier and faster to create your own script to meet your own expectations.

The skills gap between testers and developers

In most testing teams, the roles can be divided into two. The tester, and the automation technician.

The tester, in short, takes care of the definition of what is to be tested, the creation of test scripts (not to be confused with scripting test automation), along with the quality of the tests and the effectiveness of the testing.

The automation technician, in short, takes care of the efficiency of the testing and creates the automated processes.

In some cases, this role is assumed by one person. In others, it can be divided between several.

When you couple this with the fact that the skills required from professionals are changing, problems start to arise. Businesses are looking for profiles with experience in development and testing. A team with multi-skilled personnel is attractive for obvious reasons – your team will be able to deliver tasks faster, and more cost-effectively. However, not many of these multi-skilled professionals are available. And those that are available are a costly resource.

Adopting tools that enable non-technical users to automate therefore closes the skills gap that will inevitably arise in most test automation projects.

Related reading: How No-code Test Automation Closes the Skills Gap in Software Development

Resisting test automation

Typically, you will experience pushback from non-technical testers (i.e., functional testers, business analysts, etc.) when introducing test automation. This reaction can (in some cases) whittle down to two reasons:

1. They fear it will take over their job
2. They have a very heavy workload and do not see the value in creating automated tests if it will not immediately relieve their workload

If you’re facing the first dilemma, it is important to reassure your team that test automation is not here to replace jobs. It will take away the boring tasks like regression testing, so they can focus on the important and challenging tests.

Now for the second dilemma. If your testers are very busy testing the system, they will feel as though there is no time to take on additional projects. If the automation tool takes too long to learn because it is script-based, it is likely that your team will not adopt it. This will make it very difficult to scale automation beyond siloed projects. Forcefully “spoon-feeding” the solution will not make the problem go away.

The path to least resistance

In this case, it is important to adopt an automation solution that has a short onboarding time and is easy to use for business users and testers. Code-based solutions make this difficult (see reasons above), while visual solutions that don’t require coding experience make it easier and faster for the entire team to build, reuse and maintain tests.

As we covered in the “Why you should automate testing in SAP” section, not all test environments are equal, and the requirements will differ greatly from one company to the next.

That being said, there are common threads that appear across most organizations in terms of the basic functions a test automation solution for SAP should have. The following are listed below:

• A short learning curve, enabling testers to set up test cases from day one
• Easy-to-use, enabling anyone to set up, maintain and run tests
• Scalable, allowing you to run your entire regression suite at high speed and frequency

Related reading: How to Increase Your SAP Regression Testing Coverage

• Cross-technology automation is possible, working across all web, mobile, and desktop apps and technologies
• Keeps a record, capturing why test cases fail in videos and logs
• Reliable, keeping your data and most business-critical processes safe
• Supported, getting you help when you need it

By adopting codeless automation, you can increase the quality of your software and in the process, better prepare for updates and changes to your SAP system. Whether it’s for specific use cases, like planning tables used in SAP APO, or to serve as a replacement for SAP’s own testing tools.

With a tool like Leapwork, you aren’t dependent on programming knowledge of SAP’s proprietary language – ABAP. Instead, you can tap into the potential of those testing your system – the business users.

Learn more about Leapwork for Testing in SAP

Listed below are some of the capabilities of Leapwork for SAP:

• It’s easy to use and adopt. Business users and QA managers can take ownership of SAP automation. With the fastest learning curve on the market, testers are equipped to automate in less than three months.
• Mitigate risk at scale. Quickly ramp up automation coverage to ensure end-to-end business process continuity across your organization.
• Say goodbye to heavy maintenance. With hyper-visual debugging tools, maximum reusability with reusable components, and data-driven capabilities, maintenance is extremely efficient, no matter how complex your SAP system requirements. No coding skills needed.
• You can release with quality SAP customizations and implementations at speed. Accelerate the delivery of SAP updates, modifications, and service pack rollouts and release with confidence.
• Users can collaborate seamlessly. Enable people with different skills and expertise to work efficiently together and automate SAP.
• Migrations to S/4HANA. You can protect against downtime with Leapwork’s rapid automation of your testing. You’ll achieve maximum coverage of your system so that you can migrate your system with confidence. With a network of consulting partners, you can deliver automation faster. Read our solution brief on Leapwork for S/4HANA migrations.
• Automate across all technologies. Whether your application integrates with multiple web-based portals, or you need to run data-driven tests for third-party customer success tools, we’ve got you covered. With Leapwork, you can use the same visual approach to build automation across all your integrated applications.

You can watch a demonstration of codeless automation in action in SAP in our on-demand webinar.