Blog

Introduction

My name is Finn Fiedler, and I am a Software Engineer at smapiot, a company specialized in delivering IT solutions and services in the space of Internet of Things, Identity and Access Management, and Portal Solutions.

Over the last few years, I worked on multiple projects also as a Quality Assurance Engineer. There, I was building test frameworks from scratch and reengineering existing ones. Furthermore, I defined many test cases for complex scenarios.

In this article, I will share my experiences of performing reliable E2E testing for large-scale projects. What are its advantages, what degree of E2E testing is too much, and why writing E2E tests can be a very satisfying experience.

Although the examples in this article are focused on E2E testing that involve a UI the majority of the points also account for other applications as well.

Audience

This article is dedicated to Quality Assurance (QA) teams thinking of automating their testing processes. It would help if you have a general understanding of what E2E testing is (automated or manual) and also what is meant by "selectors" (e.g., CSS selectors).

Reason for automated E2E testing

The best thing about end-to-end tests is that they directly represent a customer's interaction. It is straightforward to prove the quality of a product or feature to business people, product owners, and customers with a well-defined E2E test.

A single test can cover a wide range of components. It validates all required parts of the system for a successful user journey. That includes UI components, backend services, network, database configuration, etc.

There are multiple E2E testing frameworks in the market, but they all usually have one thing in common: They are pretty easy to use. That makes writing E2E tests very easy for QA-Engineers who do not have a software development background.

Challenges of E2E tests

1) E2E tests are very fragile and hard to maintain

Compared to other types of tests like unit or integration tests, E2E tests need a lot more attention. They might very suddenly stop working for various reasons even if the feature is doing fine.

Two main reasons for that could be:

1. Changes to test data. For example, if the test user has access to some asserted resources, but some service now delivers different resources, the test might fail. These changes can be communicated to the QA's before they go live, but you cannot rely on catching everything in large co-operations.
2. Changes to the UI. If some element is wrapped inside a new div element, that could already break a selector of your test. Even if the user does not see any visible changes and the logic is not broken, the test might fail.

One problem is that you also do not know the cause of the error straight away, which leads to the next downside of E2E tests.

2) The root cause of an error is not straight visible

A failing E2E test always requires a lot of investigation. Because so many systems are involved in a single test run, it can be hard to figure out what component in the chain caused the actual error. They also may fail randomly for network errors, external dependencies, simple non-visible changes, etc. Maybe it was a database issue, or a Kubernetes pod was rescheduled just in time the request happened. But perhaps it is indeed a regression error, and your test caught it. The big challenge is not to get sloppy here; even if 90% of the time, when the test case fails, the feature is working fine.

3) Handling of Secrets

Since E2E tests heavily rely on secrets, you have to find a way to manage that problem. You can read more in the section below.

Tips for creating an excellent E2E test strategy

1) Make each test a mirror of a user journey

When defining a test case, you can follow the same naming convention as defining user stories in agile projects (“As an ADMIN I would like to ...”).

An example of a well-named test case could be: “A French customer can find his machine by using the local search bar.” This test case covers a realistic scenario that a real customer might take, and the naming also clarifies the intention. With this naming convention, the value of the test cases also becomes clear to Product owners and Stakeholders.

A bad example would be a test case like: “Verify that table headers are ordered correctly.” Whoever views this test case does not know immediately why a failure of this test case would block the user journey.

2) General software engineering principles also apply for writing E2E tests

The most important one to mention here is the DRY (“Don't repeat yourself”) principle. This can be achieved by using the Page-objects pattern. Simply spoken, you can group your selectors for each application page and then use these page objects within your tests.

If your application is huge and your project uses a component library, you can even go further and create an interaction class for each of these components. You can then place these interaction components in your page objects directly.

In the following example, we have an accordion table component that is heavily used within the application. The interaction component has a root selector that is received in the constructor. We can now add methods to interact with these components. In the following example, we create the method:

expandOrCloseRow(index: number)

We can now place these component interactors inside our page object files and like this:

At first, this might seem a little complicated, but it impressively scales in a testing project with hundreds or even thousands of automated E2E tests.

3) Create maintainable selectors when your tests involve UI testing

Well-created selectors are a crucial success factor for an excellent E2E testing project. In general, you should try to keep selectors as short as possible.

Try to avoid dynamically created selectors. For example, if an admin user and a customer have access to the same button but have a different selector to access it (e.g., because the admin has an additional banner displayed on top). You can use unique attributes on your relevant nodes to avoid functions that generate selectors based on logic.

The following code shows a best-practice example tagging each testable element with the attribute "test-anchor-id".

The UI component above can be accessed with the simple CSS selector:

[test-anchor-id=”trader-fx-market-selection”]

4) Less is more

Since E2E tests are very fragile and need regular maintenance, you should keep a fixed amount of test cases related to your team's number of QA engineers (e.g., 10 test cases per engineer). If you automate too much, the maintenance expense will become too massive, and your QA's cannot concentrate on assuring quality. But they will be instead busy all the time with investigating failed test cases.

Approach for managing test secrets

Most applications involve some form of authentication. That means that our automated tests have access to a test user or multiple test users and their related authentication secrets (passwords). That can happen in two ways:

1. You can place a secret store as a JSON file directly in the testing project containing all the test users your project needs. Please make sure to exclude it from your version control, as uploading secrets is a bad practice! Every QA working on the test project then needs to place the file in his project or even create his own test users if he is privileged.
2. You can create a test user microservice in your company's environment. Your test project can then authenticate with a single secret to that service. Whenever a test is executed it can request the test user's credentials and related test data from the service by providing a synonym and the environment the test is running against.

The benefit of this is that the tests can be executed against multiple environments. They do not have to be written again for each environment because the test framework handles the resolution for them.

Test Execution

E2E tests only have a real benefit if they are executed once in a while or regularly. Here are two approaches:

1) You can run the tests together in a team

This is a handy approach if your test project is small (more then 100 test cases). A good time for that would be the daily meeting of your QA team. You can execute the tests, discuss possible failures immediately and think of strategies to tackle them.

2) CI pipeline execution

You want to go for this approach in a larger project with 100 or more test cases. You execute all your tests in a CI pipeline and check the pipeline's artifacts. These Artifacts can be screenshots or video recordings of the failing test cases but definitely should contain a file with test metrics.

You could run the pipeline every night so the QA team could check the results every morning. Suppose you have the resources to advance the pipeline. In that case, you can also enable advanced reporters in the pipeline, like email routers or Slack reporters, that publish the results in the relevant channels.

Summary

Automated E2E tests are an excellent method to constantly prove the quality of a software product to stakeholders and customers. If you focus on writing maintainable tests that your team can manage, they can benefit the project considerably.