Appearance
3.2 — Structured Outputs
1. What is it?
Structured output is the practice of forcing an LLM's response to conform to a predefined schema — a fixed JSON shape, an enum of allowed values, a specific set of fields with specific types — instead of free-form natural language text. Instead of asking the model to write a paragraph and hoping you can parse an answer out of it, you ask it to produce, for example, exactly {"vendor_name": "Acme Corp", "total": 150.0, "line_items": [...]} every single time, in a form your code can json.loads() and trust.
This chapter goes deep into how this is actually enforced under the hood, because "just ask for JSON" and "the API guarantees valid JSON matching your schema" are two very different levels of reliability, and knowing which one you're getting from a given provider/technique changes how you build the rest of your system around it.
2. Why does it exist?
LLMs are, by default, free-text generators (Part 2.6) — nothing about next-token prediction inherently produces well-formed JSON, let alone JSON matching a specific schema with the right field names and types. Left alone, a model asked to "extract the invoice total as JSON" will often comply well, but will also sometimes wrap the JSON in explanatory prose ("Sure! Here's the extracted data: json ..."), use a slightly different field name than you expected, output a number as a string, or occasionally produce subtly malformed JSON (a trailing comma, an unescaped quote). Every one of these failure modes breaks a downstream json.loads() call or a database write.
Structured output tooling exists to close this gap between "the model usually gets the format right" and "the format is enforced, every time, provably."
3. What problem does it solve?
It solves the fundamental integration problem between a non-deterministic, natural-language system (the LLM) and a deterministic, typed system (everything else in your application — your database, your business logic, your downstream APIs). Without structured output, every LLM call that needs to feed into code requires brittle, ad-hoc text parsing (regexes, string splitting, hoping the model didn't add a stray sentence). With it, an LLM call becomes a typed function call from your code's perspective: you get back exactly the shape you asked for, or an explicit, catchable failure.
This is the single technology that makes it possible to treat "call an LLM" as just another function in your codebase rather than a special, fragile, hand-parsed integration point — and it's the direct technical foundation underneath tool/function calling (Part 3.3) and every production agent architecture in this book.
4. How does it work internally?
There are three genuinely different mechanisms in use across the industry, and they give different strength of guarantee. Understanding which one a given provider/library is actually using matters enormously for how much you trust the output without additional validation.
Mechanism 1: Prompting alone (weakest guarantee)
The simplest approach: describe the desired JSON shape in the prompt ("respond only with JSON matching this schema: ...") and hope the model complies. This relies entirely on the model's trained tendency to follow formatting instructions — there is no enforcement at the decoding level. Modern, well-aligned models are quite good at this, but "quite good" is not "guaranteed," and at production scale (thousands or millions of calls), even a 99.5% compliance rate means a steady stream of malformed outputs your code must handle.
Mechanism 2: Constrained decoding / grammar-based generation (strongest guarantee)
This is what's actually happening under the hood for most providers' "guaranteed structured output" features (e.g., JSON mode / structured outputs features that promise schema-conformant output). Recall from Part 2.6 that generation works by the model producing a probability distribution over the entire vocabulary for the next token, then sampling from it. Constrained decoding intervenes at exactly this step: before sampling, it computes which tokens are even grammatically valid given the schema and what's been generated so far (based on the schema, translated into something like a finite-state grammar), and masks out (sets to zero probability) every token that would violate the schema. The model can only ever sample a token that keeps the output on a path toward valid, schema-conforming JSON.
Schema requires{"category": "billing" | "technical" | "account"}
Model has generated so far{"category": "bi
Next-token distributionfull vocabulary — thousands of possible tokens
Constrained decoding maskseverything except tokens that continue toward "billing"
Effective distribution["lling"] ≈ 100% — only valid continuation
This is why this mechanism gives a genuinely strong guarantee: it is structurally impossible for the model to emit a token that breaks the schema, because that token's probability was zeroed out before sampling ever happened. It's not "the model was told to behave and mostly does" — it's "the invalid option was never on the table." This is a meaningfully different (and stronger) guarantee than prompting alone, and it's worth knowing whether a given API feature you're using actually does this at the decoding level, versus doing something closer to mechanism 3.
Mechanism 3: Generate, validate, retry/repair (a pragmatic middle ground)
Some tools (and some fallback paths even in "guaranteed" APIs) work by generating output somewhat freely, validating it against your schema (e.g., with Pydantic), and if it fails validation, either automatically retrying with an error message appended to the prompt ("your previous output failed validation because X, please fix and resend") or attempting to programmatically repair small issues (e.g., stripping markdown code fences, fixing a trailing comma). This is weaker than true constrained decoding (the invalid output was actually generated once, then caught) but is more portable across models/providers that don't support decoding-level constraints, and is genuinely effective in practice when combined with a low retry budget and good error messages fed back to the model.
Function/tool calling as a special case of structured output
When you give a model a tool definition with a JSON Schema for its arguments (Part 3.3), the model producing a "tool call" is, under the hood, the exact same structured-output mechanism applied to the tool's argument schema instead of a general-purpose schema you defined yourself. This is why understanding structured output deeply is a direct prerequisite for understanding tool calling, agents, and MCP — they're all built on this same foundation, layered with more orchestration on top.
5. Simple mental model
Think of unconstrained generation as writing a check with no restrictions — the model can write anything, and you find out afterward whether it cashed correctly. Constrained decoding is like a form with physical cutouts in the paper — a fill-in-the-blank form where the blanks are literally shaped like the only valid answers, so it's physically impossible to write something that doesn't fit. Generate-validate-retry is like writing the check, having someone check it before it goes to the bank, and handing it back with a red pen mark if it's wrong — slower, and the "wrong" version briefly existed, but it still catches the problem before it reaches anything that matters.
6. Real-world example
A logistics company's AI extracts shipment details from unstructured emails: carrier name, tracking number, estimated delivery date. Early in the project, the team used prompting alone ("respond in JSON with these fields") and found that roughly 1 in 200 emails produced output wrapped in explanatory text or with a date in an inconsistent format ("March 5" vs "2026-03-05"), silently breaking their downstream scheduling system on those emails. Switching to a provider's schema-enforced structured output mode (constrained decoding) with the date field typed and formatted via the schema itself eliminated the malformed-JSON failures entirely, and adding a Pydantic validator with a strict date parser caught and flagged the remaining semantic issues (a nonsensical or ambiguous date) as an explicit, loggable validation error instead of a silent bad write.
7. Architecture diagram
Your code
LLMconstrained decoding against the schema
Pydantic validationyour typed boundary
Notice there are two distinct layers here: schema conformance (is this valid JSON with the right field names/types — often enforced by the provider) and semantic validity (is total_amount_usd: -50.0 actually a sensible value — never enforced by the provider, always your responsibility). Never assume the first layer makes the second layer unnecessary.
8. Production considerations
- Always validate structured output with your own typed models (Pydantic) even when using a provider's schema-enforcing feature. The provider guarantees syntactic conformance (right shape, right types) — it does not and cannot guarantee semantic correctness (a
total_amount_usdof-50.0is syntactically valid JSON matching your schema and semantically nonsensical). - Keep schemas as simple as the task allows. Deeply nested, highly conditional schemas are harder for constrained decoding to satisfy well and harder for a model to reason about correctly — flatten where you can.
- Version your schemas alongside your prompts (Part 3.1, Part 1.7) — a schema change is a contract change for every downstream consumer of that structured output.
- Design for the retry/repair path explicitly even when using constrained decoding, because constrained decoding guarantees syntactic validity, not that the model chose the semantically best values — a validation failure on the semantic layer still needs a defined retry or fallback behavior.
9. Common mistakes
- Trusting a provider's "structured output" feature to mean "the values are correct," when it only means "the shape is correct."
- Skipping Pydantic (or equivalent) validation because "the API already guarantees JSON," missing semantic errors entirely.
- Writing an overly complex schema (deeply nested optional fields, many conditional branches) and then being confused when output quality is inconsistent — schema complexity has a real cost on model reliability, not just a cosmetic one.
- Not handling the case where structured output generation itself fails or times out (a real possibility under provider-side issues), leaving no fallback path.
10. Security considerations
- Structured output does not make LLM output trusted input — a syntactically valid JSON object can still contain adversarial or malicious values (e.g., a string field containing a SQL injection payload, or a tool-call argument requesting deletion of unrelated resources). Schema conformance is a data-shape guarantee, not a content-safety guarantee (Part 9.1, Part 9.2).
- When structured output feeds directly into a tool call with real side effects (Part 3.3), the semantic validation layer (business-rule checks: is this account ID one the current user is authorized to touch) is a security control, not just a data-quality one.
11. Performance considerations
- Constrained decoding can add a small amount of latency overhead per token (computing the valid-token mask), though this is typically negligible compared to model inference time itself for most schemas — pathologically large or complex schemas can be an exception, worth benchmarking if you notice unexpected latency.
- A generate-validate-retry approach adds a full extra round trip (and cost) on every validation failure — worth tracking your failure rate in production as a real cost/latency metric (Part 8), not just a correctness one.
12. Cost considerations
- Retries on validation failure cost additional tokens/calls — a schema that produces frequent validation failures is a direct, measurable cost problem, giving you a concrete incentive (beyond correctness) to simplify a poorly-performing schema.
- Complex schemas with verbose field descriptions add to the prompt's token count on every single call (the schema itself is typically included in the request) — keep descriptions clear but not redundant.
13. When to use it
Any time an LLM's output needs to be consumed by code rather than read directly by a human — which is the overwhelming majority of AI engineering integration points: extraction tasks, classification, tool/function calling, any API endpoint returning LLM-derived data.
14. When NOT to use it
- Purely conversational, human-facing responses (a chat reply meant to be read as natural prose) don't benefit from forcing a rigid schema and would be actively harmed by it (stilted, unnatural output).
- Extremely open-ended creative or exploratory generation tasks, where the value is in not constraining the model's output space.
15. Alternatives and trade-offs
| Approach | Guarantee strength | Good for | Weak point |
|---|---|---|---|
| Prompting alone ("respond in JSON") | Weak — no enforcement | Quick prototypes, low-stakes tasks | Real failure rate at scale, needs manual parsing/handling |
| Constrained decoding (provider structured output feature) | Strong — syntactically guaranteed | Production extraction, tool calling, any code-consumed output | Doesn't guarantee semantic correctness; schema complexity has real limits |
| Generate-validate-retry | Medium — caught, not prevented | Providers/models without constrained decoding support, or as a semantic-layer backstop | Extra latency/cost per retry; invalid output briefly exists |
16. Practical Python/code example
python
from pydantic import BaseModel, Field, field_validator
from anthropic import AsyncAnthropic
client = AsyncAnthropic()
class InvoiceExtraction(BaseModel):
"""Structured fields extracted from an invoice document."""
vendor_name: str
invoice_number: str
total_amount_usd: float = Field(gt=0)
@field_validator("invoice_number")
@classmethod
def not_empty(cls, v: str) -> str:
"""Rejects an empty invoice number — a semantic check no schema alone catches."""
if not v.strip():
raise ValueError("invoice_number must not be empty")
return v
async def extract_invoice(document_text: str) -> InvoiceExtraction:
"""
Extracts structured invoice fields using schema-enforced generation, then
independently validates the result with Pydantic for semantic correctness.
Args:
document_text (str): Raw invoice text to extract from.
Returns:
InvoiceExtraction: The validated, structured extraction result.
"""
response = await client.messages.create(
model="claude-sonnet-4-5",
max_tokens=500,
tools=[
{
"name": "record_invoice",
"description": "Records extracted invoice fields.",
"input_schema": InvoiceExtraction.model_json_schema(),
}
],
tool_choice={"type": "tool", "name": "record_invoice"},
messages=[{"role": "user", "content": f"Extract invoice fields from:\n{document_text}"}],
)
tool_use_block = next(b for b in response.content if b.type == "tool_use")
return InvoiceExtraction.model_validate(tool_use_block.input)Note the two-layer pattern from section 7: input_schema=InvoiceExtraction.model_json_schema() gets the provider's schema-enforcement working from the shape defined once in the Pydantic model (single source of truth), and InvoiceExtraction.model_validate(...) re-validates independently, catching the semantic rules (gt=0, the custom validator) that the JSON Schema passed to the model doesn't fully express as strictly.
17. Production-quality example
Adding the retry-with-repair layer (mechanism 3) as a backstop for the rare case where semantic validation fails even after schema-enforced generation:
python
import logging
from pydantic import ValidationError
logger = logging.getLogger("structured_extraction")
class ExtractionFailedError(Exception):
"""Raised when structured extraction fails validation after all retry attempts."""
async def extract_invoice_with_retry(
document_text: str, max_attempts: int = 2
) -> InvoiceExtraction:
"""
Extracts and validates invoice fields, retrying with the validation error fed
back to the model if the first attempt fails semantic validation.
Args:
document_text (str): Raw invoice text to extract from.
max_attempts (int): Maximum number of extraction attempts before giving up.
Returns:
InvoiceExtraction: The validated extraction result.
Raises:
ExtractionFailedError: If validation still fails after max_attempts.
"""
last_error: ValidationError | None = None
correction_note = ""
for attempt in range(1, max_attempts + 1):
response = await client.messages.create(
model="claude-sonnet-4-5",
max_tokens=500,
tools=[
{
"name": "record_invoice",
"description": "Records extracted invoice fields.",
"input_schema": InvoiceExtraction.model_json_schema(),
}
],
tool_choice={"type": "tool", "name": "record_invoice"},
messages=[
{
"role": "user",
"content": f"Extract invoice fields from:\n{document_text}{correction_note}",
}
],
)
tool_use_block = next(b for b in response.content if b.type == "tool_use")
try:
return InvoiceExtraction.model_validate(tool_use_block.input)
except ValidationError as exc:
last_error = exc
correction_note = (
f"\n\nYour previous extraction failed validation: {exc}. "
"Please correct it and try again."
)
logger.warning("extraction attempt %d failed validation: %s", attempt, exc)
raise ExtractionFailedError(f"extraction failed after {max_attempts} attempts: {last_error}")This is the concrete pattern behind "generate-validate-retry": the validation error becomes part of the next prompt, giving the model the specific, actionable information it needs to self-correct — far more effective than a generic "please try again."
18. Short exercise
Take the InvoiceExtraction model and add a line_items: list[str] field with a validator requiring at least one item. Then trace through, step by step, what happens if the model returns line_items: [] when using (a) schema-enforced generation alone, and (b) schema-enforced generation plus your Pydantic validation layer. Explain exactly which layer catches the problem and why the other one doesn't.
19. Interview questions
- Explain constrained decoding at the token-probability level, and why it gives a stronger guarantee than prompting the model to "respond in JSON."
- Why is schema conformance not sufficient on its own, and what specific class of errors does a semantic validation layer (Pydantic) catch that schema enforcement cannot?
- Design the retry strategy for a structured-extraction pipeline where roughly 2% of calls fail semantic validation — what would you feed back to the model on retry, and how would you bound the retry budget?
20. FDE/customer scenario
Customer: "Your extraction pipeline occasionally puts a negative number in a field that should never be negative — I thought you said the output was validated?"
This is the two-layer distinction from section 7 in the wild. The honest, technically precise answer: the provider's structured-output feature guarantees the shape (a number in that field, not a string, not missing) — it does not and structurally cannot guarantee the value makes business sense. The fix is adding (or fixing a gap in) the Pydantic-layer semantic validation (e.g., Field(gt=0)), not asking the LLM provider to "try harder" — this is a genuinely common point of confusion worth proactively explaining to customers before it becomes a trust problem.
Key takeaways
- Structured output has (at least) three implementation mechanisms with meaningfully different guarantee strength: prompting alone (weak), constrained decoding (strong, syntactic), generate-validate-retry (medium, pragmatic).
- Schema conformance (shape/type correctness) and semantic validity (value correctness) are two different, independently necessary layers — never assume one implies the other.
- Tool/function calling (Part 3.3) is structured output applied to a tool's argument schema — same mechanism, more orchestration on top.
Things you should be able to explain
- How constrained decoding works at the token-probability-masking level.
- Why a schema-enforced API guarantee doesn't make LLM output "trusted" in the security sense.
Things you should be able to build
- A two-layer extraction pipeline: schema-enforced generation plus independent Pydantic validation, with a retry-with-feedback loop for semantic failures.
Common mistakes
- Treating schema conformance as a proxy for correctness.
- Skipping independent validation because "the API already guarantees JSON."
- Overly complex, deeply nested schemas that degrade reliability.
Recommended next chapter
03-tool-function-calling.md