E2E Tests on a Real Backend: Signup Starting from Email

Until October, the E2E tests in the BusiKM web panel sat in the repository and guarded nothing. The CI workflow was disabled for pull requests, and its header had a comment explaining why: the lack of stable data on staging produced a flood of false alarms, and 111 tests times three browsers with retries took over 20 minutes and usually ended in a timeout. The tests ran only after a push to main, which is when it was already too late to react. In October I turned them into a gate for PRs to develop and added a test that was previously impossible to write: a full signup with an activation email.

Why not mocks#

The fastest way to green tests would have been to mock the backend in Playwright. The tests would always pass, but they would say nothing about whether the web app and the backend actually work together. And that is exactly where most things break, especially when the API contract changes every few days. I chose the other path: tests on a real backend, but in an environment prepared for testing, not on staging with random data.

There are three changes. The E2E environment has its own accounts, created with the seed_e2e command: a regular company with confirmed accounts, not a demo account. The tests run in a single browser, chromium, just like in local verification, so a run is no longer multiplied by three. And the workflow is switched on by a repository variable, so PRs don't turn red before the environment has test accounts.

A test that checks whether the company accepts writes#

The demo account in BusiKM works in read-only mode, so a test that writes anything will bounce off it with a 402 error. Locally the tests run on demo accounts, in CI on accounts from seed_e2e. Instead of guessing what a test is running against, the helper asks the backend directly:

/** Stops the test if the company does not accept writes: an empty POST gives 400 (validation) or 402. */
export async function requireWritable(request: APIRequestContext): Promise<void> {
  const probe = await request.post("/api/proxy/v1/counterparties/", { data: {} });
  if (probe.status() !== 402) return;
  if (REQUIRE_WRITES) throw new Error(READ_ONLY_REASON); // CI gate: read-only is an error
  test.skip(true, READ_ONLY_REASON); // locally: skip
}

An empty POST won't pass validation, so it won't save anything. A 400 response means writes work, and a 402 means the company is read-only. In the CI gate the latter is an environment misconfiguration, while locally it's a reason to skip the test.

An email with nowhere to receive it#

The registration test was the hardest. The user fills in the form, gets an email with an activation link, confirms the account, logs in, picks a plan and lands on the dashboard with a checklist. Without the email this path can't be completed, and the E2E environment has no inbox that Playwright could access.

The solution is a separate app in the backend, test_support. Its email backend intercepts messages to addresses from the test domain and stores them in the database instead of sending them. Messages to any other address go out the usual way.

class CapturingEmailBackend(BaseEmailBackend):
    """Mail to the E2E domain is stored in the database, the rest goes on as usual."""
 
    def send_messages(self, email_messages):
        captured = [m for m in email_messages
                    if m.recipients() and all(map(is_e2e_address, m.recipients()))]
        delivered = [m for m in email_messages if m not in captured]
        for message in captured:
            CapturedEmail.objects.bulk_create(
                CapturedEmail(recipient=r, subject=message.subject,
                              body=message.body, html_body=_html_body(message))
                for r in message.recipients()
            )
        sent = 0
        if delivered:
            connection = get_connection(settings.E2E_DELEGATE_EMAIL_BACKEND)
            sent = connection.send_messages(delivered) or 0
        return len(captured) + sent

There are two endpoints: one returns the latest message to an address along with the links extracted from it, the other deletes the test account after the test. The first has a contains filter, because two emails go out after signup, the activation one and the welcome one, and the "latest" happens to be the welcome one. The second deletes the company only if it has no other users, because the test account could have been added to someone else's company.

The Playwright test itself polls this endpoint in a loop for 30 seconds until the email appears. Each run creates a separate company with a random but valid NIP (Polish tax ID) with a correct checksum, so tests don't collide, and the account disappears after the test.

Test code in production code#

Endpoints that hand out someone's mail and delete accounts sound like asking for trouble in application code. That's why they have several layers of protection:

• they work only where E2E support is enabled in the settings, and the production settings disable it permanently,
• when disabled, they respond with 404, so as not to reveal that such endpoints exist at all,
• they require an environment token in the header, compared with a hmac.compare_digest function that is resistant to timing attacks,
• they accept only addresses from the test domain, so they can't be used to read or delete a real account.

The mailbox endpoint has no rate limit, because the test polls it in a loop. Access requires a token anyway, so a limit wouldn't add protection here, it would only break the test.

The same in the driver app#

The mobile app uses the same infrastructure. A Maestro smoke test builds a release version on an Android emulator. Before the test, the owner invites a driver through the API, and the invitation code is read from the email captured by the backend. The test goes through the welcome screen, entering with the invitation code, creating the account, permission consents, and a route without an assignment from start to confirmation. The next steps from the release checklist, namely the receipt, offline mode and password reset, are a separate task for later.

What it costs#

The E2E environment is one more environment to maintain, with its own accounts and secrets. The E2E support token must be the same in the environment's backend and in the CI secrets of both the web and mobile apps. If it's missing, the registration test is skipped rather than reporting an error, so you have to remember that a skipped test is not a green test.

The second cost is test code in the backend repository. The alternative would be a separate mail-capturing service, but that's one more thing to set up, maintain and expose to CI. The test_support app has its own tests and is disabled wherever it shouldn't run.

The most important change wasn't technical. Before, the E2E tests checked screens, and now they check the path a new customer takes: from the form, through the email, to the first screen in the app. If that path breaks, nobody will pay for the rest of the features. If you want to talk about testing in your SaaS, write to me.