1.9 — Testing

1. What is it?

Testing is the practice of writing code that verifies your code behaves correctly, automatically, repeatably. In AI systems, "testing" splits into two genuinely different disciplines that are easy to conflate: traditional software testing (unit/integration tests for your deterministic code — API handlers, DB queries, business logic) and AI evaluation (measuring the quality of non-deterministic LLM/agent output — covered in depth in Part 8). This chapter covers the former; Part 8 covers the latter; a mature AI system needs both, and they use different tools for good reasons.

2. Why does it exist?

Without automated tests, verifying correctness means manually re-checking behavior every time code changes — which doesn't scale past a trivial codebase and doesn't survive a team growing past one person. Tests exist to make regressions loud and immediate (a CI failure) instead of silent and delayed (a customer reporting a bug days later).

3. What problem does it solve?

For AI systems specifically: the non-deterministic parts (LLM outputs) can't be tested with a plain assert result == expected_value — but the deterministic scaffolding around them absolutely can and must be: does your retrieval function return the right documents for a known query, does your tool-calling code correctly parse a well-formed function-call response, does your API return 401 for an unauthenticated request. Skipping this because "the AI part is fuzzy anyway" is a common and costly mistake — most production AI bugs are in the deterministic plumbing, not the model.

4. How does it work internally?

The testing pyramid, applied to AI systems

E2Efew, slow, expensive — full agent run against a real/staging LLM + vector DB, asserts on final behavior

Integrationmore, moderate cost — DB queries, API endpoints, mocked LLM calls

Unitmany, fast, cheap — pure functions: prompt formatting, parsing, validation, retrieval ranking math

Width in a classic pyramid = quantity; a fourth, parallel evaluation-suite track (Part 8) runs alongside all three for the non-deterministic parts.

The classic testing pyramid still applies, with one AI-specific addition: a fourth, parallel track of evaluation suites (Part 8) that run against real or recorded model outputs to measure quality/regression — these are neither unit nor integration tests in the traditional sense (they don't assert exact equality) but they play the same "catch regressions automatically" role for the non-deterministic parts.

Mocking the LLM boundary

from unittest.mock import AsyncMock

async def test_extraction_parses_llm_response(monkeypatch):
    """Verifies parsing logic handles a well-formed LLM tool-call response correctly."""
    mock_llm = AsyncMock(return_value={"tool_calls": [{"name": "extract", "args": {"total": 42.5}}]})
    result = await parse_extraction_response(mock_llm.return_value)
    assert result.total_amount_usd == 42.5

This tests your parsing code, not the LLM — which is exactly right for a unit test. You cannot (and should not try to) unit-test "does the LLM correctly extract the total from an invoice" with a plain assertion; that's an evaluation problem (Part 8), because the LLM's actual output varies and "correct" is often graded, not exact-matched.

Testing an async FastAPI route with httpx.ASGITransport

TestClient (Part 1.3) is synchronous under the hood and works for most cases, but testing an async def route in its own async context — or running it alongside other pytest-asyncio async fixtures — calls for httpx.AsyncClient wired directly to the app via ASGITransport, with dependency_overrides swapping the real DB/LLM clients for test doubles:

import httpx
import pytest
from myapp.main import app
from myapp.dependencies import get_llm_client, get_db_session


class FakeLLMClient:
    """A test double standing in for the real async LLM client."""

    async def post(self, *args, **kwargs) -> dict:
        """Returns a fixed, canned response instead of calling a real model."""
        return {"output": "mocked summary"}


@pytest.fixture
def override_dependencies():
    """Swaps real DB/LLM dependencies for test doubles for the duration of one test."""
    app.dependency_overrides[get_llm_client] = lambda: FakeLLMClient()
    app.dependency_overrides[get_db_session] = lambda: FakeDBSession()
    yield
    app.dependency_overrides.clear()


@pytest.mark.asyncio
async def test_summarize_route_returns_mocked_output(override_dependencies):
    """
    Exercises the real async route end-to-end (routing, validation, dependency
    injection) while never making a real LLM API call or touching a real database.
    """
    transport = httpx.ASGITransport(app=app)
    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
        response = await client.post("/v1/summaries", json={"text": "a long document"})

    assert response.status_code == 200
    assert response.json()["summary"] == "mocked summary"

ASGITransport sends requests directly into the app's ASGI callable in-process — no real socket, no running server — so this is still a fast, hermetic test, just one that exercises the actual async request path (middleware, Depends() resolution, response serialization) rather than calling the handler function directly. Clearing dependency_overrides after each test (the fixture's yield/cleanup) prevents one test's fake client from silently leaking into the next test.

Fixtures and test isolation

import pytest
import pytest_asyncio

@pytest_asyncio.fixture
async def test_db_session():
    """Provides an isolated, rolled-back DB session for each test."""
    async with test_engine.begin() as conn:
        async with AsyncSession(bind=conn) as session:
            yield session
            await conn.rollback()

Each test getting a clean, isolated environment (a rolled-back transaction, a fresh mock) is what makes tests trustworthy and order-independent — a test suite where tests pass or fail depending on execution order is worse than no test suite, because it erodes trust in the whole signal.

5. Simple mental model

Testing is a tripwire net under a construction site: you don't test to prove the building is perfect, you test so that when something breaks, it breaks loudly and immediately, at the point of the change that caused it — not three floors up, three weeks later, discovered by whoever's standing underneath (in this analogy: your customer).

6. Real-world example

A team ships a refactor of their RAG retrieval function that accidentally reverses the similarity ordering (returns least-relevant chunks first). Their retrieval unit test (assert results[0].id == expected_top_chunk_id against a fixed, known embedding set) fails immediately in CI — the bug never reaches a customer-facing agent, and never gets confused for "the LLM just isn't very good today," which is what would have happened without the test isolating the deterministic retrieval step from the LLM's own variability.

7. Architecture diagram

CI pipeline · every PR

Unit testsfast, no network

Integration testsDB, mocked LLM

Eval suitereal/recorded model output · Part 8

Unit and integration tests fail fast and block merge; the eval suite runs separately.

8. Production considerations

• Run unit + integration tests on every PR in CI; run the (typically slower, sometimes costlier) evaluation suite on a schedule and/or before releases, not necessarily on every commit.
• Mock external LLM/API calls in unit tests for speed and determinism; reserve real calls for a smaller set of integration/eval tests, since real LLM calls are slow, cost money, and are non-deterministic (making assertions flaky if used carelessly).
• Test failure modes explicitly: what does your code do when the LLM API times out, returns malformed JSON, or returns a refusal instead of the expected structured output? These are common production incidents and are exactly the kind of thing that should be tested, not discovered live.
• Golden/regression tests for prompts: a fixed set of inputs with known-good expected characteristics (not always exact match — sometimes a rule like "must contain a phone number" or a score threshold from an LLM-judge), re-run whenever the prompt changes.

9. Common mistakes

• Testing only the "happy path," never the LLM timeout/malformed-output/rate-limit paths that are common in production.
• Confusing "the LLM said something different" with "the code is broken" — without separating deterministic unit tests from evaluation, teams either write brittle exact-match tests against LLM output (constantly flaking) or give up on testing the AI-adjacent code entirely.
• Slow test suites (tests that make real network calls) that developers start skipping locally, eroding the whole practice.
• Not testing authorization/tenant-isolation logic — a bug here isn't just a functional bug, it's a security incident (Part 9).

10. Security considerations

• Explicitly test authorization boundaries: a test that asserts "user from tenant A cannot retrieve tenant B's documents" is one of the highest-value tests in a multi-tenant AI system, and one of the easiest to skip because it's not "core feature" work.
• Test injection-adjacent parsing/validation logic (Part 9.1) — e.g., that a malformed or adversarial tool-call argument is rejected by your validation layer, not passed through.

11. Performance considerations

• Keep the unit test suite fast (seconds, not minutes) so it's actually run locally before every commit — slow suites get skipped, which defeats the purpose.
• Use test parallelization (pytest-xdist) for larger suites once integration tests start adding up.

12. Cost considerations

• Real LLM calls in a large, frequently-run test suite are a real, recurring cost — budget for it deliberately (e.g., a smaller eval set run on every PR, a larger one run nightly) rather than accidentally discovering a large API bill from CI.

13. When to use it

Always, for the deterministic parts of any production AI system: API contracts, data validation, retrieval logic, tool execution, authorization, error handling.

14. When NOT to use it

• Don't try to unit-test exact LLM output text with equality assertions — that's the wrong tool (use evaluation, Part 8) and produces a flaky, low-trust suite.
• Extremely early-stage prototypes exploring feasibility may reasonably defer a full test suite briefly — but this should be a deliberate, temporary trade-off, not a permanent habit that survives into production.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Unit tests (pytest)	Fast, deterministic logic verification	Can't meaningfully assert on LLM output quality
Integration tests	Verifying real component interaction (DB, API)	Slower, more setup/teardown complexity
LLM evaluation (Part 8)	Measuring non-deterministic output quality	Not a replacement for deterministic correctness testing
Manual QA	Catches things automated tests miss (UX feel)	Doesn't scale, doesn't run on every commit

16. Practical Python/code example

import pytest
from myapp.extraction import parse_invoice_response, InvalidExtractionError


def test_parse_invoice_response_valid():
    """Valid, well-formed LLM tool-call output parses into the expected structured model."""
    raw = {"vendor_name": "Acme Corp", "invoice_number": "INV-001", "total_amount_usd": 150.0}
    result = parse_invoice_response(raw)
    assert result.vendor_name == "Acme Corp"
    assert result.total_amount_usd == 150.0


def test_parse_invoice_response_rejects_empty_invoice_number():
    """Malformed output (empty invoice number) raises rather than passing through silently."""
    raw = {"vendor_name": "Acme Corp", "invoice_number": "", "total_amount_usd": 150.0}
    with pytest.raises(InvalidExtractionError):
        parse_invoice_response(raw)


def test_parse_invoice_response_rejects_missing_fields():
    """Missing required fields raise a clear validation error, not a KeyError deep in the code."""
    with pytest.raises(InvalidExtractionError):
        parse_invoice_response({"vendor_name": "Acme Corp"})

17. Production-quality example

A tenant-isolation test — arguably the single highest-value test category in a multi-tenant AI system:

import pytest
import pytest_asyncio
from myapp.vector_store import TenantScopedVectorStore


@pytest_asyncio.fixture
async def seeded_store(test_db_session):
    """Seeds two tenants' document chunks with distinguishable content."""
    store = TenantScopedVectorStore(session_factory=lambda: test_db_session)
    await seed_chunk(test_db_session, tenant_id="tenant-a", content="Tenant A confidential data")
    await seed_chunk(test_db_session, tenant_id="tenant-b", content="Tenant B confidential data")
    return store


@pytest.mark.asyncio
async def test_tenant_isolation_in_vector_search(seeded_store):
    """
    A search scoped to tenant-a must never return tenant-b's documents,
    regardless of embedding similarity.
    """
    results = await seeded_store.search(
        tenant_id="tenant-a", query_embedding=SIMILAR_TO_BOTH_EMBEDDING, k=10
    )
    assert all("Tenant B" not in r["content"] for r in results)
    assert any("Tenant A" in r["content"] for r in results)

This test is deliberately written to fail loudly if a future refactor accidentally drops the tenant_id filter — exactly the kind of regression that's catastrophic in production and invisible without this test.

18. Short exercise

Write a test (pseudocode is fine) that verifies your agent's tool-execution code correctly rejects a tool call with an argument that fails your pydantic schema validation, without ever actually executing the tool's side-effecting logic.

19. Interview questions

• Why is it a mistake to unit-test exact LLM output text with equality assertions, and what should you do instead?
• Describe the difference in purpose between an integration test and an evaluation suite for an AI system.
• What's the highest-value test category you'd prioritize first in a multi-tenant AI SaaS product, and why?
• How does httpx.ASGITransport differ from spinning up a real server for testing a FastAPI route, and why does dependency_overrides matter for keeping that test hermetic?

20. FDE/customer scenario

Customer's security reviewer: "Prove to us that one customer's data can never leak into another customer's AI responses before we sign off on production deployment."

The credible answer is not "we're confident in our code" — it's "here is our automated tenant-isolation test suite, run on every commit, and here's the specific test that would fail if that isolation were ever broken." Being able to point at a concrete, running, CI-enforced test is often what actually satisfies this kind of review, versus a verbal assurance that satisfies no one.

---

Key takeaways

• Deterministic code (parsing, retrieval, auth) gets traditional tests; non-deterministic LLM output gets evaluation (Part 8) — conflating the two produces either flaky tests or untested AI code.
• Tenant-isolation tests are disproportionately high-value in multi-tenant AI systems.

Things you should be able to explain

• Why exact-match assertions on LLM output are the wrong tool.
• The testing pyramid as applied to an AI system, including where evaluation fits.

Things you should be able to build

• A test suite covering happy-path, malformed-input, and tenant-isolation cases for an AI-adjacent service.
• An async FastAPI route test using httpx.ASGITransport and dependency_overrides to swap out real DB/LLM clients.

Common mistakes

• Only testing the happy path.
• Flaky tests from real, uncontrolled LLM calls in unit tests.
• Skipping authorization/tenant-isolation tests.

Recommended next chapter

Part 1 complete. Continue to handbook/02-ai-fundamentals/01-ml-basics.md.