Documentation/Test frameworks & CI/GitLab CI

Email testing in GitLab CI

The same per-run inbox pattern in .gitlab-ci.yml, with the key stored as a masked CI/CD variable and teardown in after_script.

How it fits together

after_script runs whether the job succeeded or failed, which makes it the right place for teardown, the GitLab equivalent of if: always(). Two GitLab details decide whether this works on the first try. after_script runs in a fresh shell, so a variable exported in before_script is gone by the time you need it for the delete call, and the address has to travel through a dotenv artifact instead. And a retried job keeps the same CI_PIPELINE_ID, so naming the inbox after the pipeline means the second attempt asks for an address the first attempt already took.

The job

CI_JOB_ID rather than CI_PIPELINE_ID: it changes on every attempt, so a retry opens its own box instead of colliding with the one the first attempt left behind.

YAML

# .gitlab-ci.yml
e2e:
  image: mcr.microsoft.com/playwright:v1.61.0-jammy
  variables:
    MAILFLAT_API: "https://mailflat.net/api/v1"
  before_script:
    - apt-get update && apt-get install -y jq
    - |
      export TEST_EMAIL=$(curl -s -X POST "$MAILFLAT_API/inboxes" \
        -H "X-API-Key: $MAILFLAT_API_KEY" \
        -H "Content-Type: application/json" \
        -d "{\"prefix\":\"ci-$CI_JOB_ID\",\"retention_hours\":2}" | jq -r .address)
      echo "TEST_EMAIL=$TEST_EMAIL" >> build.env
  script:
    - npm ci
    - npx playwright test
  after_script:
    - |
      curl -s -X DELETE "$MAILFLAT_API/inboxes/$TEST_EMAIL" \
        -H "X-API-Key: $MAILFLAT_API_KEY" || true
  artifacts:
    reports:
      dotenv: build.env

Worth knowing

Add MAILFLAT_API_KEY under Settings, CI/CD, Variables and tick Masked so it never appears in job logs. Leave Protected off unless every branch that runs this job is protected, because a protected variable is simply absent on ordinary branches and the job then sends no key at all.

after_script runs in a fresh shell, so pass the address through a dotenv artifact rather than an exported variable.

after_script has its own timeout, separate from the job's. A teardown that waits on anything slow can be cut off while the job itself is still reported as passing, which is how inboxes quietly accumulate.

retention_hours is the safety net under all of this. Even if the delete never runs, the messages expire on their own, and on Free that happens two hours later.

A scheduled pipeline is the one case where a fixed address beats a fresh one. Put the address in a CI/CD variable, give the job a key with inbox:read only, and nothing in the pipeline is allowed to create or delete anything. The address is permanent, so it is still there next week.

When it goes wrong

Symptom	Why	What to do
The create step returns 403 on a merge request from a fork	Project CI/CD variables are not passed to pipelines from forked projects, so MAILFLAT_API_KEY is empty and the request goes out unauthenticated. The same happens on an unprotected branch when the variable is marked Protected.	Echo the length of the variable, never the value, as the first line of the job. A length of zero says the variable never arrived, which is a very different problem from a key that is wrong.
A retried job fails with the address already being taken	CI_PIPELINE_ID is the same on every attempt, so ci-$CI_PIPELINE_ID asks for the exact address the previous attempt created. Addresses here are permanent, so the first one is still sitting there.	Name the box after CI_JOB_ID, which is unique per attempt, or delete it in after_script so the name frees up. This is the failure that only shows up the first time someone hits Retry.
The delete in after_script does nothing and no error is printed	TEST_EMAIL was exported in before_script and after_script runs in a new shell, so the variable is empty and the URL ends in a slash with nothing after it.	Write the address to build.env and declare it as a dotenv report artifact, as in the job above. Then check the teardown log for the actual address rather than trusting that curl exited zero.
The fourth job of a matrix build cannot open an inbox	Free allows three agent inboxes at once and a matrix opens one per parallel job. This is a plan allowance, not a rate limit, and the answer is a 400 that names both the plan and the number.	Delete each box at the end of its own job so the allowance frees up, or run the matrix against one shared address. Paid plans do not cap agent inboxes.
A nightly schedule finds the inbox empty even though yesterday's run passed	The inbox survived, the messages did not. Retention is two hours on Free, so mail sent by yesterday's run is long gone by the time tonight's job reads it.	Trigger the mail inside the job that reads it. If a job genuinely needs to inspect older mail, paid plans hold messages for up to thirty days.

Email testing in GitLab CI

How it fits together

The job

Worth knowing

When it goes wrong

See also