Module 10 · Testing · Deep Dive

Testing Python & FastAPI

From "it works on my machine" to a green test suite you trust — pytest, fixtures, mocking, async, and testing every endpoint of DocChat without ever calling a real LLM.

BasicIntermediateBuild

1 · Why test at all

In PHP you may have leaned on manual clicking, a few var_dumps, and the prayer that QA caught the rest. Automated tests replace that ritual with a button you press that proves your code still does what it claimed. The payoff is not "fewer bugs" in the abstract — it's confidence to refactor.

A suite that runs in two seconds means you can rip apart the messy function you wrote last month, restructure it cleanly, and instantly know whether you broke anything. Without tests, that refactor is a gamble nobody takes, so the codebase rots. Three reasons tests earn their keep:

• Regression safety — yesterday's fixed bug stays fixed because a test guards it.
• Living documentation — a test named test_ask_returns_404_for_unknown_doc tells the next developer exactly how the endpoint behaves.
• Design pressure — code that's hard to test is usually badly coupled. The pain is a signal, not an annoyance.

2 · pytest basics

The whole Python ecosystem standardises on pytest 8. Install it and write a test as an ordinary function whose name starts with test_:

# pip install pytest

# app/math_utils.py
def add(a: int, b: int) -> int:
    return a + b

tests/test_math_utils.py

from app.math_utils import add

def test_add_two_positives():
    assert add(2, 3) == 5

def test_add_with_zero():
    assert add(5, 0) == 5

Note the plain assert — no special methods. pytest rewrites assertions so that on failure it prints both sides of the comparison ("assert 6 == 5"), which is why you never need assertEqual. Test discovery is automatic: pytest finds files matching test_*.py, then functions named test_* inside them.

Run from your project root:

# Run everything, quietly
pytest

# Verbose — show each test name and PASS/FAIL
pytest -v

# Stop at the first failure (fast feedback loop)
pytest -x

# Run only tests whose name matches a keyword
pytest -k "add and zero"

# A single file or a single test
pytest tests/test_math_utils.py::test_add_with_zero

-k is your scalpel during development: while building the ask endpoint, run pytest -k ask -x and ignore the other 200 tests until that one is green.

3 · Fixtures & conftest

A fixture is reusable setup. Instead of building the same sample document in ten tests, you build it once in a fixture and pytest injects it wherever a test names it as a parameter:

import pytest

@pytest.fixture
def sample_doc():
    return {"title": "Intro", "text": "the cat sat"}

def test_doc_has_title(sample_doc):   # name matches the fixture → injected
    assert sample_doc["title"] == "Intro"

Fixtures that hold resources (a database connection, a temp file) often need teardown. Use yield: everything before it is setup, everything after runs when the test finishes — pass or fail.

@pytest.fixture
def db_session():
    session = SessionLocal()        # setup
    yield session                   # hand it to the test
    session.rollback()              # teardown — always runs
    session.close()

Scope controls how often a fixture rebuilds. Default is function (fresh per test — safest). Widen it only for expensive, read-only setup:

@pytest.fixture(scope="session")   # built once for the whole run
def embedding_model():
    return load_model()            # slow — don't repeat it 300 times

The scopes, narrowest to widest: function, class, module, package, session. Wider scope is faster but riskier — a mutable fixture shared across tests lets one test pollute another.

4 · Parametrize

Don't copy-paste a test five times to check five inputs. @pytest.mark.parametrize runs one test body against many cases, each reported separately:

@pytest.mark.parametrize("text, expected", [
    ("hello", 1),
    ("hello world", 2),
    ("", 0),
    ("  spaced  out  ", 2),
])
def test_word_count(text, expected):
    assert len(text.split()) == expected

That's four independent tests from one function. When the empty-string case fails, pytest tells you precisely which row broke: test_word_count[-0]. The discipline this teaches is to think in edge cases — empty input, whitespace, the boundary value — because the parametrize list is a checklist of "what could go wrong".

5 · Mocking — and when to mock

A mock is a stand-in object that records how it was called and returns whatever you tell it. You reach for one when the real thing is slow, costs money, is non-deterministic, or simply isn't available in CI. For DocChat that means exactly one category: the LLM call and the embedding API.

The simplest tool is pytest's built-in monkeypatch, which temporarily replaces an attribute and restores it automatically after the test:

def test_summary_uses_llm(monkeypatch):
    # Replace the real LLM call with a fixed answer
    monkeypatch.setattr(
        "app.rag.call_llm",
        lambda prompt: "a fake summary",
    )
    assert summarize("long text") == "a fake summary"

monkeypatch also swaps environment variables and dict keys — perfect for forcing a test config:

monkeypatch.setenv("OPENAI_API_KEY", "test-key")

For richer control — asserting how something was called — use unittest.mock. MagicMock is a chameleon object; patch swaps a name for the duration of a block:

from unittest.mock import patch, MagicMock

def test_ask_calls_llm_once():
    fake = MagicMock(return_value="answer")
    with patch("app.rag.call_llm", fake):
        ask("what is RAG?")
    fake.assert_called_once()                  # it ran exactly once
    args, kwargs = fake.call_args
    assert "what is RAG?" in args[0]      # prompt contained the question

6 · Testing async & FastAPI

DocChat's endpoints are async def, so a plain test function can't await them. Install pytest-asyncio and mark the test:

# pip install pytest-asyncio
import pytest

@pytest.mark.asyncio
async def test_embed_async():
    result = await embed_text("hello")
    assert len(result) == 1536

Now the FastAPI app itself. For synchronous route testing, the classic TestClient (built on httpx) still works and reads cleanly — it spins the app up in-process, no server, no network:

from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

def test_health():
    resp = client.get("/health")
    assert resp.status_code == 200
    assert resp.json() == {"status": "ok"}

For genuinely async tests — where you want to exercise the app's async stack end to end — the modern approach is httpx AsyncClient with ASGITransport. The transport wires httpx directly into your ASGI app, no live socket:

import pytest
from httpx import AsyncClient, ASGITransport
from app.main import app

@pytest.mark.asyncio
async def test_health_async():
    transport = ASGITransport(app=app)
    async with AsyncClient(transport=transport, base_url="http://test") as ac:
        resp = await ac.get("/health")
    assert resp.status_code == 200

Whichever you use, the assertions are the same shape: check status_code first, then dig into resp.json(). Always assert the status code — a 422 validation error returns valid JSON too, and a test that only checks the body can pass against the wrong response.

7 · Overrides & a test database

DocChat's routes depend on a database session through FastAPI's Depends(get_db). In tests you don't want to touch production Postgres. FastAPI gives you a clean seam: app.dependency_overrides — swap any dependency for a test double, app code untouched.

from app.main import app
from app.db import get_db

def override_get_db():
    yield test_session            # a session pointed at the test DB

app.dependency_overrides[get_db] = override_get_db
# ... run tests ...
app.dependency_overrides.clear()   # reset so overrides don't leak

The same trick swaps out auth — override get_current_user to return a fixed test user and you skip real token plumbing in every endpoint test.

What database should the override point at? Two common strategies:

• Transactional rollback — wrap each test in a transaction and roll it back at teardown. The DB is identical before and after, so tests don't pollute each other and you skip slow re-seeding. This is the gold standard for speed and isolation.
• A real ephemeral Postgres via testcontainers — spin up a throwaway Postgres in Docker for the test run.

@pytest.fixture
def test_session(engine):
    connection = engine.connect()
    txn = connection.begin()            # open a transaction
    session = Session(bind=connection)
    yield session
    session.close()
    txn.rollback()                      # undo everything the test did
    connection.close()

8 · Build: testing DocChat

First the shared setup in conftest.py — a transactional session, a client with the DB overridden, and a fixture that mocks both external calls:

tests/conftest.py

import pytest
from fastapi.testclient import TestClient
from app.main import app
from app.db import get_db

@pytest.fixture
def client(test_session):
    def override_get_db():
        yield test_session
    app.dependency_overrides[get_db] = override_get_db
    yield TestClient(app)
    app.dependency_overrides.clear()   # never leak between tests

@pytest.fixture
def fake_ai(monkeypatch):
    # Deterministic embedding: a fixed 1536-dim vector
    monkeypatch.setattr("app.rag.embed_text",
                        lambda text: [0.1] * 1536)
    # Deterministic LLM: echoes a canned answer
    monkeypatch.setattr("app.rag.call_llm",
                        lambda prompt: "RAG retrieves then generates.")

Now the tests. Notice the arrange-act-assert rhythm and that every external call is faked:

tests/test_endpoints.py

def test_upload_document(client, fake_ai):
    # Act
    resp = client.post("/documents", json={
        "title": "Intro to RAG",
        "text": "Retrieval augmented generation explained.",
    })
    # Assert: status first, then the body
    assert resp.status_code == 201
    body = resp.json()
    assert body["title"] == "Intro to RAG"
    assert "id" in body

def test_ask_returns_grounded_answer(client, fake_ai):
    # Arrange: a document must exist to retrieve against
    client.post("/documents", json={
        "title": "Doc", "text": "RAG explained."})
    # Act
    resp = client.post("/ask", json={"question": "What is RAG?"})
    # Assert
    assert resp.status_code == 200
    assert resp.json()["answer"] == "RAG retrieves then generates."

def test_ask_validates_empty_question(client, fake_ai):
    resp = client.post("/ask", json={"question": ""})
    assert resp.status_code == 422   # Pydantic v2 rejects it

Run it: pytest tests/test_endpoints.py -v. Three green tests, no money spent, no flakiness. That last test — checking the 422 — proves your Pydantic v2 validation works, which is exactly the kind of edge case interviewers probe.

9 · Coverage & the TDD loop

coverage.py (via the pytest-cov plugin) tells you which lines your tests actually ran:

# pip install pytest-cov
pytest --cov=app

# Branch coverage — did you test BOTH sides of every if?
pytest --cov=app --cov-branch --cov-report=term-missing

--cov-report=term-missing prints the exact line numbers you never touched — your to-do list. Aim for roughly 80–90% on business logic; chasing 100% wastes time on trivial getters. Branch coverage is the honest metric: 100% line coverage can still miss the else branch you never exercised, and that's where bugs hide.

10 · Check yourself

Answer from memory — retrieval is what moves this from "I read it" to "I know it".