Testing

Cheatsheet #

What each layer is for #

The pyramid is about feedback speed, not virtue. A test that takes ten minutes to tell you something a unit test could say in ten milliseconds is a worse test even when it is more realistic.

Test behaviour through the public interface. Tests coupled to internals fail on every refactor and pass through real regressions, which is the worst of both.

A good test #

def test_rejects_expired_token():
    # arrange
    token = make_token(expires_at=datetime(2020, 1, 1, tzinfo=UTC))

    # act
    result = verify(token, now=datetime(2024, 1, 1, tzinfo=UTC))

    # assert
    assert result == Err(TokenExpired)

Name the test after the behaviour: rejects_expired_token, not test_verify_2. The name is what you read in a CI log at 3am.

Test doubles #

Mock at the boundary you own — your UserStore interface, not the database driver three layers down. Mocking a third-party client’s internals produces tests that pass while the integration is broken.

def test_sends_notification(monkeypatch):
    sent = []
    monkeypatch.setattr(notifier, "send", lambda msg: sent.append(msg))

    deploy(version="1.4.2")

    assert sent == ["deployed 1.4.2"]

A fake in-memory implementation of your own interface usually beats a mock: it exercises the same contract for every test that uses it and fails loudly when the contract changes.

Fixtures and data #

@pytest.fixture
def db(postgres_container):                       # session-scoped container
    conn = connect(postgres_container.dsn)
    with conn.begin() as tx:                      # each test in a transaction
        yield conn
        tx.rollback()                             # rolled back: no cleanup code, no leakage

@pytest.mark.parametrize(
    ("raw", "expected"),
    [("1s", 1), ("2m", 120), ("1h30m", 5400)],
    ids=["seconds", "minutes", "compound"],
)
def test_parse_duration(raw, expected):
    assert parse_duration(raw) == expected

Build test data with a factory that takes overrides, so each test states only what it cares about:

def make_user(**over):
    return User(**{"id": 1, "email": "a@example.com", "active": True, **over})

Integration tests #

Run the real dependency, not an approximation of it. Containers make that cheap and repeatable.

from testcontainers.postgres import PostgresContainer

@pytest.fixture(scope="session")
def postgres_container():
    with PostgresContainer("postgres:17") as pg:
        run_migrations(pg.get_connection_url())
        yield pg

docker compose up -d --wait        # wait for healthchecks, not sleep 30
pytest tests/integration
docker compose down -v

Test the HTTP surface through the application’s own router rather than a live socket where possible — same code path, no port allocation, no flakes:

client = TestClient(app)
r = client.post("/items", json={"name": "widget"})
assert r.status_code == 201
assert r.json()["id"]

Contract tests #

A consumer records what it needs; the provider verifies it can still supply it. This catches the break at the provider’s build instead of in staging.

pact-broker publish ./pacts --consumer-app-version "$(git rev-parse --short HEAD)"
pact-provider-verifier --provider-base-url http://localhost:8080 --pact-broker-base-url "$BROKER"

Schema checks are a cheaper approximation: generate the OpenAPI or protobuf schema in CI and fail the build if it changed incompatibly.

End-to-end #

test('user can check out', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();
  await expect(page.getByText('Order confirmed')).toBeVisible();   // auto-waits
});

Select by role and accessible name, never by CSS class. Never use fixed sleeps: assert on the condition and let the framework wait. Every flaky end-to-end test is a race between the test and the application, and disabling it is usually admitting the application has one too.

Coverage and mutation #

Coverage shows which lines ran, not whether anything was verified. It is a useful floor (uncovered code is definitely untested) and a useless target (100% coverage with weak assertions proves nothing).

pytest --cov=src --cov-report=term-missing --cov-fail-under=80
go test -coverprofile=c.out ./... && go tool cover -func=c.out | tail -1
npx vitest run --coverage
mutmut run                     # mutation testing: does a test fail when the code is wrong?

Mutation testing answers the question coverage cannot: it changes the code and checks a test notices.

Flakes #

pytest -p no:cacheprovider --count=20 tests/test_flaky.py     # pytest-repeat
go test -count=100 -race -run TestFlaky ./pkg

Quarantining a flaky test hides a real defect about half the time. Fix it or delete it; a test nobody trusts is worse than no test.

In CI #

Run the fast layers on every push and the slow ones before merge. Fail on the first layer that fails, and make the report say which test and which assertion without anyone opening a browser.

ruff check . && mypy --strict src/ && pytest -q --cov=src --cov-fail-under=80
go vet ./... && go test -race -count=1 ./...
npm run lint && npx tsc --noEmit && npx vitest run --coverage

-count=1 in Go disables the test cache, which is what you want in CI and not what you want locally.