Documentation/Test frameworks & CI/GitHub Actions

Email testing in GitHub Actions

Give every workflow run its own inbox, pass the address to the suite as an environment variable, and sweep it up even when the job fails.

How it fits together

The pattern is the same as locally, with two CI-specific details: the key comes from repository secrets, and teardown runs under if: always() because that is exactly when inboxes get left behind. The failure worth designing against is quieter than either of those. A curl that is refused still exits zero, jq turns the error body into the string null, and the suite then runs against an address that reads literally as null, so every test fails on a timeout waiting for mail. The cause is three steps earlier and nothing in the log says so.

The workflow

Two details make the quiet failure loud: curl --fail turns a refusal into a non zero exit, and the guard on the address stops the job before the suite spends its timeout budget on an address that was never created. GITHUB_RUN_ATTEMPT is in the name because a re-run keeps the same run id, and the address the first attempt created is still there.

YAML

# .github/workflows/e2e.yml
name: e2e
on: [push]
 
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      MAILFLAT_API_KEY: ${{ secrets.MAILFLAT_API_KEY }}
      MAILFLAT_API: https://mailflat.net/api/v1
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
 
      - name: Open an inbox for this run
        run: |
          ADDRESS=$(curl -sS --fail-with-body -X POST "$MAILFLAT_API/inboxes" \
            -H "X-API-Key: $MAILFLAT_API_KEY" \
            -H "Content-Type: application/json" \
            -d "{\"prefix\":\"ci-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}\",\"retention_hours\":2}" | jq -r .address)
          test -n "$ADDRESS" -a "$ADDRESS" != "null" || { echo "inbox was not created"; exit 1; }
          echo "TEST_EMAIL=$ADDRESS" >> "$GITHUB_ENV"
 
      - run: npm ci
      - run: npx playwright test
 
      - name: Tear down
        if: always()
        run: |
          curl -s -X DELETE "$MAILFLAT_API/inboxes/$TEST_EMAIL" \
            -H "X-API-Key: $MAILFLAT_API_KEY" || true

Or let the suite do it

If your tests already create an inbox per test, the workflow only needs the key.

YAML

      - run: npx playwright test
        env:
          MAILFLAT_API_KEY: ${{ secrets.MAILFLAT_API_KEY }}

Worth knowing

retention_hours is the safety net: even if the teardown step never runs, the mail expires on its own. On Free that happens two hours later.

An account key printed into a public build log grants access to every inbox you own. Keep it in secrets.

Secrets are not passed to workflows triggered by a pull request from a fork. The job still runs, the key is an empty string, and the create step is refused. Echo the length of the key rather than the key, because a length of zero is a completely different problem from a key that is wrong.

GITHUB_ENV reaches later steps in the same job and nothing else. A teardown placed in a separate job sees no address at all, which is why the cleanup above sits in the same job under if: always().

Free allows three agent inboxes at once. A matrix build opens one per leg, so a four way matrix is refused on its fourth leg with a 400 that names the plan. This is a plan allowance, not a rate limit.

The address is permanent until you delete it. For a nightly smoke run, keeping one address in a repository variable and giving the job a key with inbox:read only is both simpler and safer than creating and deleting on every run.

When it goes wrong

Symptom	Why	What to do
Every test times out waiting for mail and the address in the log reads null	The create call was refused, curl still exited zero, and jq rendered the error body as the string null. TEST_EMAIL then held null and the suite waited for mail at an address that does not exist.	Use curl --fail-with-body and assert the address is neither empty nor null, as above. Without that guard the real error is three steps earlier and the log never mentions it.
The workflow passes on branches and fails on pull requests from forks	Secrets are withheld from fork pull requests, so MAILFLAT_API_KEY is empty and the request goes out unauthenticated.	Skip the mail steps on fork events, or move them to a workflow that runs after review. Printing the key's length as the first line makes this obvious in one glance.
Re-running a failed job fails on the create step	GITHUB_RUN_ID is unchanged by a re-run, so the prefix asks for the address the first attempt already created. Addresses are permanent, so it is still taken.	Add GITHUB_RUN_ATTEMPT to the prefix as shown, or make sure the teardown ran the first time round. This is the failure that only appears the first time someone presses Re-run.
Teardown reports success while inboxes keep accumulating	The delete is in a different job, so TEST_EMAIL is empty there and the URL ends with nothing after the slash. The trailing \|\| true then swallows whatever the server said.	Keep teardown in the same job under if: always(), and print the address it is deleting rather than trusting the exit code.
The create step returns 403 with a valid key	Opening a box needs the inbox:manage scope and a newly issued key carries inbox:read only. A key that is scoped to a single inbox is also refused here, whatever its scopes, because a new box would fall outside what it is allowed to reach.	Issue an account level key with inbox:manage for the workflow, or drop the create step and run against a fixed address held in a repository variable.