End-to-End Tests: Why They Are Expensive and How to Reduce Costs

Automated end-to-end tests validate critical business processes, but they can be expensive to build, run, and maintain. Explore the main cost drivers and practical strategies for reducing maintenance, execution, and failure analysis costs.

Automated end-to-end tests are complex and costly. At the same time, their importance is undisputed, because only this type of testing can validate the correctness and completeness of real-world business processes that span multiple systems.

In practice, however, creating, maintaining, and updating automated end-to-end tests is often labor-intensive. Their execution is also time-consuming and frequently unstable because of the large number of systems involved. Even small changes in the test environment, including the underlying infrastructure, the applications under test, and other connected systems involved in the business process, can cause tests to fail or abort.

Martin Fowler summarizes this problem succinctly in his description of the Test Pyramid:

“In short, tests that run end-to-end through the UI are: brittle, expensive to write, and time consuming to run.”

Source: Martin Fowler, Test Pyramid

These costs do not arise in just one area. They are distributed across the creation and maintenance of automated tests, test environments and connected systems, test data management, test execution itself, and the analysis of failed tests.

Why Automated End-to-End Tests Are Expensive

The costs of automated end-to-end tests are driven primarily by:

• the creation, maintenance and update of automated end-to-end tests,
• the provision, maintenance, and upkeep of the test environment and other connected systems involved in the business process,
• the provision and maintenance of test data (test data management),
• long test execution times,
• the risk of unstable test executions, and
• the high effort required to analyze failed test executions.

Other factors that affect the cost and value of automated end-to-end tests include:

• the complexity of the overall system,
• the planned lifespan of business-critical applications, and
• organizational responsibilities for automated end-to-end tests.

The following sections examine these individual cost factors in more detail. Where helpful, practical examples illustrate the dependencies involved.

Creating, Maintaining, and Updating Automated End-to-End Tests

At the beginning of an automation project, a test automation specialist, referred to below as an “automation engineer,” often creates an automated end-to-end test for a core business process. This usually happens only after the technical feasibility and business viability have been successfully evaluated in a proof of concept (PoC).

Because the automation engineer generally does not have all of the required domain knowledge, they depend on subject matter experts to understand the business process correctly, implement it as an automated end-to-end test, and provide suitable test data.

Maintenance work resulting from operating system, browser, or test automation tool updates is unavoidable, but it should remain manageable.

As the number of automated end-to-end tests grows, so does the effort to update the tests. Changes to the applications under test often require updates to multiple test cases. Application changes usually result from business requirements or regulatory requirements. Changes to third-party interfaces, such as data formats, can also require adjustments.

From a technical perspective, the automation engineer needs a test automation tool such as Playwright (Microsoft), Selenium (open source), UFT Functional Testing (OpenText), or Tosca (Tricentis), often in combination with a test management tool or test platform such as Jira (Atlassian) or ALM (OpenText).

The resulting costs can vary significantly, ranging from free open-source solutions to commercial test automation tools and platform combinations that can result in annual costs in the five-figure range per automation engineer.

Cost factors:

• Personnel costs: automation engineers, subject matter experts, developers, and other members of the test team or organization
• Hardware costs: depending on the test platform and test automation tool used
• License costs: for paid products used to create, manage, orchestrate, and execute automated tests

Test Environment and Connected Systems

Testing business processes with end-to-end tests generally requires a dedicated test environment that mirrors the production environment as closely as necessary for the planned tests. Provisioning, maintaining, and supporting this test environment and its connected systems is therefore often handled by corporate IT, which is already highly familiar with the production system and the applications running on it.

When tests are scheduled in this environment, the applications under test must be at the required version level, and the test environment, including all connected systems, must be operational when testing begins.

When complex test platforms are used, their operation requires specialized expertise. An OpenText ALM installation, for example, consists of several components (ALM Application Server, license server, database server, and file storage) and must support specialized functions such as remote test execution. These components are generally administered and operated by specially trained employees, often within corporate IT.

Cost factors:

• Personnel costs: corporate IT, test management, developers
• Hardware costs: for an internally hosted test environment
• License costs/fees: for paid products or the use of external systems

Test Data Management

In practice, the effort involved in test data management often proves to be an underestimated cost factor in end-to-end testing. Identifying suitable and correct test data can be very time-consuming and generally requires deep domain knowledge. Because this often goes beyond the automation engineer’s expertise, knowledgeable subject matter experts frequently need to be involved.

Because the test data required for end-to-end tests is often production-like, it generally needs to be prepared before it is provided and used to comply with legal requirements, such as the GDPR, as well as company-wide policies.

When end-to-end tests are used over an extended period, test data maintenance requires particular attention because, for example, the relevance of the test data can change over time.

Example from practice:

For an automated end-to-end test of the “fixed-line contract for new customers” business case, two test variants were created based on an analysis of production data: one for the standard plan and another for the plan selected by most customers.

The test case variant with the highest customer preference was assigned a “high” priority to ensure that it would be executed in every test phase.

These tests continue to run without any apparent issues for an extended period. However, they have lost relevance for the following reasons:

• a new standard plan following the introduction of a new product portfolio
• changed customer preferences following the transition from analog to IP-based telephony and the expansion of the product portfolio to include high-bandwidth fiber plans

Cost factors:

• Personnel costs: automation engineers, subject matter experts, developers
• Hardware/software costs: may arise, for example, from the use of additional database capacity, specialized test data management tools, or external services

Long Test Execution Times

In addition to the need for a production-like test environment, long execution times are a major reason why end-to-end testing is either performed entirely outside the CI/CD process or run within it only for selected builds.

Execution time depends on several factors, and the business case itself often imposes a minimum runtime.

Cost factors:

• Personnel costs: automation engineers
• Hardware costs: additional costs may arise when the test environment is billed based on usage

Unstable Test Executions

Unstable test executions create additional costs because these often long-running tests must be repeated and may need to be supplied with new test data before they are restarted.

As a result, conclusions about the correctness of the business cases can only be reached later. Testing of downstream business cases may also need to be postponed.

Beyond the direct cost impact, unstable test executions also create the risk that confidence in test automation will decline within the test team and across the organization.

Cost factors:

• Personnel costs: automation engineers, subject matter experts, developers

Analyzing Failed Test Executions

The following example illustrates why end-to-end tests can quickly generate substantial analysis effort, especially in highly complex test environments.

Example from practice:

Because of a temporary network problem, a service used in the business process is unable to reserve a phone number. Later, an application attempts to process that phone number, fails, and generates an application-specific error message.

The missing phone number initially goes unnoticed and only later causes the automated end-to-end test to terminate with a FAILED status.

If there are no indications of network problems, a developer responsible for the application in which the internal error was reported is typically brought in first to analyze the failed test run. Several analysis cycles may be required before the root cause is ultimately identified, often involving people from multiple areas.

Cost factors:

• Personnel costs: automation engineers, subject matter experts, developers, corporate IT

Additional Factors That Affect Costs

In addition to the direct costs associated with the creation, execution, and analysis of automated end-to-end tests, as well as the provision of test environments and test data, further factors influence their cost-effectiveness.

Complexity of the Overall System

The complexity of the overall system has a significant impact on the cost of end-to-end testing and affects all of the cost factors discussed above.

Example of rising maintenance costs for automated end-to-end tests:

Processing business cases in complex overall systems generally requires multiple applications. As a result, multiple automation engineers are often involved, with responsibility for the applications under test distributed among them.

Automation engineers coordinate how the automation artifacts are implemented. However, these discussions mainly concern the artifacts’ “interfaces,” meaning the input data they require and the output data they provide. The internal implementation of individual artifacts is often discussed only to a limited extent, if at all.

Possible consequences:

• UI elements are located and interacted with in different ways.
• Business logic is handled inconsistently.

With a stable team structure, these differences may initially go unnoticed. Even then, however, they affect maintenance effort because necessary application changes require varying levels of work and different approaches are used for the same or similar tasks.

When the team structure changes significantly, maintenance effort can increase sharply, limiting the team’s ability to add further test coverage.

Planned Lifespan of Business-Critical Applications

A long lifespan for business-critical applications increases the value of end-to-end tests because existing artifacts can be used for regression testing over a longer period.

At the same time, test data can lose business relevance over time, and changes in team structure can lead to higher maintenance costs.

Ownership and Responsibilities for Automated End-to-End Tests

While responsibility for automated end-to-end tests lies with the entire team, there are no universally applicable recommendations for assigning responsibility for their creation, maintenance, update, and execution.

In practice, hybrid models are common. For example, automation engineers may create and maintain the tests, while execution is handled by the people responsible for manual testing, referred to below as testers.

Key factors that can influence how responsibilities are assigned include:

• knowledge of the test platform and test automation tool in use
• customer experience and requirements
• personnel costs for automation engineers and testers

Automation engineers and testers also often differ in their domain and technical expertise. Automation engineers typically have strong technical expertise, while testers generally bring extensive domain knowledge.

How to Reduce the Cost of Automated End-to-End Tests

Understanding the factors that drive the cost of automated end-to-end tests makes it possible to identify optimization opportunities and use the clear benefits of end-to-end testing as cost-effectively as possible.

General software development practices can also help reduce costs:

• Ensuring knowledge transfer, for example through pair programming and regular code reviews
• Documenting the tests created and the test data used
• Clearly defining responsibilities

In addition, specific test automation measures can help optimize costs:

1. Select the test platform and test automation tool
2. Clearly define test objectives
3. Continuously maintain and manage test data
4. Prioritize test stability
5. Standardize and modularize test automation

Choose the Right Test Platform and Test Automation Tool

For simplicity, the term “test tool” is used below when referring generally to either a test platform or a test automation tool.

If the test tool has not yet been selected, the following considerations should be evaluated in addition to the tool’s direct cost:

• Are there company requirements, such as tool consolidation policies, or is a test tool already in use? Using an additional test tool exclusively for end-to-end testing should be avoided unless there is a clear business and economic justification.

• Are there reasons that rule out certain test tools or make a particular tool mandatory or especially appropriate? The suitability of test automation tools depends heavily on the application environment. While tools such as Playwright are designed primarily for web applications and provide only limited support for desktop applications, Tricentis Tosca offers extensive automation capabilities, particularly in SAP environments.

• Will the test platform be used only for end-to-end testing, or should it also support other automated or manual tests?

• What technical skills do users of the test tool need? While some test automation tools, such as Functional Testing, require extensive software development skills, Robot Framework’s consistent keyword-driven approach makes it a test automation tool that requires comparatively less development expertise.

• What additional capabilities must the test tool provide, for example, for documenting test executions and test results?

Define Clear Test Objectives and Limit Test Scope

End-to-end testing focuses on validating complete business processes that usually span multiple systems and applications. Because of the high effort and associated costs, the test scope should be limited to a small number of business-critical scenarios.

Running these tests requires all systems and applications involved in the respective business processes to be available in the test environment.

Reviewing individual systems and applications in light of the test objectives can reveal additional opportunities to reduce costs, particularly through:

• the use of mocking, simulation, and service virtualization techniques
• the use of API interfaces

Continuously Maintain and Manage Test Data

Providing suitable and correct test data is time-consuming and requires a high level of domain knowledge. Test data that has been identified once and reused can become “outdated” over time. As a result, tests may lose business relevance or test executions may fail.

Continuous maintenance and management of test data is therefore essential. The following measures can help:

Review and Update Test Data Regularly

Regularly reviewing test data from a business perspective is essential, particularly to maintain the business relevance of automated end-to-end tests.

Test data must be adapted to changing conditions when necessary, for example when new plans and plan options are introduced into the product portfolio or when a limit is placed on the number of mobile contracts per customer.

Automate Test Data Generation, Identification, and Validation

Automation can reduce the effort required to generate, identify, validate, and continuously maintain test data.

Examples include:

• reusing existing contracts to test current-customer processes such as contract cancellations,
• automatically identifying suitable test data using SQL statements,
• using internal or external services, and
• using tools to generate specific test data such as IBANs or SIM card numbers.

Use Test Data Management Tools

Many test tools already provide test data management capabilities. For example, test data for automated end-to-end tests created with OpenText Functional Testing can be stored centrally as test resources in the OpenText ALM test platform.

Additional tools are also available that focus specifically on test data management or support selected test management capabilities.

Prioritize Test Stability

A major cause of unstable test executions is the large number of systems, applications, and interfaces involved in the business process, along with the resulting technical dependencies.

Minimizing external influences on test execution is therefore an important way to improve test stability. In addition to the previously mentioned use of mocking and simulation techniques and API interfaces, the following measures can improve execution stability:

• ensure and continuously monitor the availability of all systems
• ensure defined initial states before every test execution
• reduce the systems involved to the components strictly required to execute the business processes

For the test automation artifacts themselves, the following measures can help improve test execution stability:

Avoid Fixed Waits

Fixed waits should be avoided whenever possible. Waits that are too long unnecessarily increase test execution time, while waits that are too short increase the risk of test failures and unstable results.

Modern test automation tools such as Playwright include built-in synchronization mechanisms, which largely eliminate the need for static waits.

If a test automation tool provides only limited synchronization support, explicit state-dependent synchronization mechanisms, such as event- or message-based approaches, should be used because they are more robust and efficient than fixed waits.

Continuously Monitor System and Application States

Because end-to-end tests involve many systems and applications, errors may not have an impact until later in the process and may ultimately cause the test to fail. This makes failure analysis more difficult and can lead to test automation scripts being modified to handle such errors independently of the business requirements.

Repeated modifications increase script complexity, reduce readability, and increase maintenance effort.

One possible countermeasure is to continuously monitor relevant system and application states during test execution.

Identify UI Elements Reliably

Ideally, a UI element can always be uniquely identified by one of its properties, such as a static ID that is valid across the application.

If this is not possible, multiple properties must be used as identification criteria. This creates the risk that an object can no longer be identified, or can no longer be identified uniquely. As a result, the UI element can no longer be located and interacted with during test execution.

For reliable UI element identification, only the element properties strictly required for unique identification should therefore be used.

Standardize and Modularize Test Automation

General software engineering practices for standardization and modularization can also be applied to the creation and maintenance of test artifacts for automated end-to-end tests:

• use standards and guidelines
• separate test logic from business logic
• provide and use reusable test artifacts

A custom test automation framework can make a significant contribution to implementing these measures.

Because designing, implementing, and introducing such a framework requires substantial time and personnel and consumes significant resources, especially at the beginning, its costs must be proportionate to the value it delivers.

When Is a Custom Test Automation Framework Worth It?

Factors that support the use of a custom test automation framework include:

• specific customer requirements
• long project durations or a long application lifespan (> 5 years)
• a high degree of test artifact reuse across multiple applications or projects
• a large number of automated tests
• short release cycles or frequent regression tests

In most cases, the decision to build a custom framework is made in long-running test automation projects and is based on the experience accumulated with test automation over the course of the project.

Rapidly increasing maintenance effort is often, though not exclusively, a key factor behind the decision to pursue a custom framework.

Conclusion: Use End-to-End Tests Selectively and Cost-Effectively

Automated end-to-end tests are particularly valuable when complete business processes spanning multiple systems need to be validated. At the same time, substantial effort is required to create, maintain and update them, provide the necessary test environments and test data, handle long execution times, and analyze failed tests.

To limit this effort, end-to-end tests should be targeted at business-critical scenarios. Appropriate tool selection, continuous test data management, high test stability, and standardized and modular test automation can also help reduce long-term costs.

The goal is not to cover as many business processes as possible completely with end-to-end tests. Instead, the value of additional test coverage should be weighed against the associated creation, execution, and maintenance effort.