Documentation/Test frameworks & CI/pytest

Email testing with pytest

A conftest fixture that gives every test its own inbox and cleans up afterwards, whether the test passed or not.

How it fits together

pytest fixtures are the natural home for this: one inbox per test, created before and deleted after, with the address handed straight to the test function. The scope you give that fixture is the whole decision. Function scope means a new box for every test. Module or session scope means one box that outlives them, and because the waiting call returns the newest message already in the box, a shared box hands the second test the first test's code and the suite stays green while proving nothing.

Skip the setup

The steps below are what a working project looks like. If you want that project rather than the explanation, generate it: one runnable smoke test, one signup test to edit, and a gitignored .env for your key.

Shell

npm create mailflat@latest -- --template pytest

Zero dependencies of its own; the generated project pulls the SDK.

The fixture

Everything after the yield is teardown, so a failing test still cleans up. Note the split: the client is session scoped because it is just an HTTP wrapper, while the inbox is left at the default function scope because that is what buys the isolation.

Python

# conftest.py
import os, uuid, pytest
from mailflat import MailFlat
 
@pytest.fixture(scope="session")
def mf():
    return MailFlat(api_key=os.environ["MAILFLAT_API_KEY"])
 
@pytest.fixture
def inbox(mf):
    box = mf.create(prefix=f"pytest-{uuid.uuid4().hex[:8]}", retention_hours=2)
    yield box
    box.delete()

The test

Python

# test_signup.py
def test_signup_sends_a_verification_code(inbox, client):
    client.post("/signup", json={"email": inbox.address})
 
    otp = inbox.wait_for_otp(timeout=60)
    assert client.post("/verify", json={"code": otp}).status_code == 200

Run it in parallel

pytest-xdist splits tests across processes. Because each test owns its inbox, nothing else has to change in the test code, but the number of boxes open at any moment now scales with the worker count, and that is what runs into the plan allowance.

Shell

pytest -n 4

Check what you actually got

A retention request above your plan maximum is capped rather than refused, and nothing is raised. The inbox object carries the value that was really applied, so one assertion in the fixture turns a silent surprise into a clear failure.

Python

@pytest.fixture
def inbox(mf):
    box = mf.create(prefix=f"pytest-{uuid.uuid4().hex[:8]}", retention_hours=24)
    assert box.retention_hours == 24, (
        f"plan capped retention to {box.retention_hours}h"
    )
    yield box
    box.delete()

Worth knowing

Keep the client session-scoped and the inbox function-scoped: one HTTP client, many inboxes.

wait_for_otp raises OTPTimeoutError, which pytest reports as a failure with the address in the message.

The Python client counts in seconds where the JavaScript one counts in milliseconds, and its waiting calls take keyword arguments only. wait_for_otp(timeout=60) waits a minute; wait_for_otp(60) raises a TypeError that has nothing to do with mail.

The 403 class is called MailFlatPermissionError rather than PermissionError, because the short name shadows a Python built-in. An except PermissionError in a module that never imported the SDK would catch the wrong one, which is the sort of bug that takes an afternoon.

pytest-timeout kills the test from outside, so its message tells you nothing about the mailbox. Keep it above the wait, or lower the wait, so the client is the one that gives up and can say whether anything arrived.

The address is permanent until you delete it. A suite that runs on a schedule can keep one address in an environment variable and use a key with inbox:read only, which removes every way for the run to delete something it should not.

When it goes wrong

Symptom	Why	What to do
A test passes with the code the previous test triggered	The inbox fixture is module or session scoped, so one box is shared. wait_for_otp returns the newest message already sitting in it and comes back instantly with a stale code.	Leave the inbox fixture at function scope, as above. Nothing in the output hints at this one, so it is worth ruling out before trusting a suite that only ever passes.
pytest -n 4 fails on some workers with a 400 about the plan	Free allows three agent inboxes at once and four workers open four boxes. This is a plan allowance rather than a rate limit, so the refusal is a 400 naming the plan and the number, not a 429.	Lower the worker count, make sure the teardown really runs so the allowance frees up, or share one address across the suite. Paid plans do not cap agent inboxes.
The fixture raises before the test body and nothing is cleaned up	Teardown after yield only runs if setup reached the yield. A create that fails leaves the fixture with nothing to delete, which is correct, but any box created earlier in the same fixture is orphaned.	Create exactly one thing per fixture and let pytest compose them. retention_hours is the backstop underneath: even an orphaned box empties itself on the plan window.
OTPTimeoutError says mail did arrive	A message landed but no code could be read out of it. That is a parsing miss, not a delivery failure, and mistaking one for the other sends you looking at DNS for an hour.	The error quotes the newest message. Read inbox.latest().text in the failure path and print it, then send us the format so it stops needing a workaround.
Everything is 403 even though the key is right	Three different situations answer with the same 403 on purpose: the address does not exist, it was deleted, or it belongs to another account. Splitting them would tell a stranger which addresses exist.	The message says all three and points at GET /api/v1/inboxes. List them and compare; if the address is not in the list, creation is what failed, not reading.

Email testing with pytest

How it fits together

Skip the setup

The fixture

The test

Run it in parallel

Check what you actually got

Worth knowing

When it goes wrong

See also