4.3 — Prompts and Structured Outputs in LangChain

1. What is it?

ChatPromptTemplate is LangChain's abstraction for parameterized, reusable prompt templates (Part 3.1's "separate the stable template from the variable content" principle, implemented as a first-class object). LangChain's structured-output support (.with_structured_output()) is its interface over the constrained-decoding and tool-calling-based mechanisms from Part 3.2 for getting schema-conformant output from a model, provider-agnostically.

2. Why does it exist?

Part 3.1 argued prompts should be versioned, parameterized, and testable like code. ChatPromptTemplate gives you an object that enforces this discipline structurally: a template with named placeholders, separate from the values filled into it at call time, composable via LCEL (Part 4.1) with a model and a parser. .with_structured_output() exists because Part 3.2 showed there isn't one universal mechanism for structured output — different providers implement schema enforcement differently (native structured-output APIs, tool-calling-based extraction, or prompting-plus-parsing) — and this method gives you one call that picks an appropriate underlying strategy for whichever provider you're using.

3. What problem does it solve?

ChatPromptTemplate solves "keep prompt structure and prompt content separate and reusable," directly enabling the versioning/testing discipline from Part 3.1/1.7/1.9. .with_structured_output() solves "get schema-conformant output without hand-writing provider-specific structured-output code for each provider you support" — abstracting over Part 3.2's mechanism differences the same way BaseChatModel (Part 4.2) abstracts over raw message format differences.

4. How does it work internally?

ChatPromptTemplate as a Runnable

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You triage tickets into: {categories}."),
    ("user", "{ticket_text}"),
])

ChatPromptTemplate implements the same Runnable interface from Part 4.1 — calling .invoke({"categories": "...", "ticket_text": "..."}) fills the placeholders and returns a list[BaseMessage] (Part 4.2), ready to be piped (|) directly into a model. Because it's a Runnable, it composes into LCEL chains uniformly, and because the template and its input variables are separate, explicit objects, you can version the template itself in Git (Part 1.7) independent of what gets filled into it at runtime — directly operationalizing Part 3.1's "version prompts like code" principle.

.with_structured_output() — what actually happens underneath

structured_model = model.with_structured_output(InvoiceExtraction)

This returns a new Runnable that, depending on the underlying provider and model, does one of: (a) uses the provider's native constrained-decoding structured-output feature if available (Part 3.2's strongest-guarantee mechanism), or (b) converts your schema into a tool definition and forces a tool call, then parses the result (structurally identical to Part 3.2's practical example using tools + tool_choice), or (c) falls back to a prompting-plus-output-parsing strategy for providers/models without either of the above. Which strategy is used for a given provider/model combination is worth verifying explicitly against current LangChain documentation — this is precisely the kind of "don't assume; the guarantee strength differs" distinction Part 3.2 emphasized, and the abstraction's convenience shouldn't obscure that the underlying guarantee strength can genuinely differ depending on what's running underneath it for your specific provider.

.with_structured_output(YourPydanticModel)

Native structured output?

Tool calling support?

Fallback: prompting + output parsingweaker guarantee (Part 3.2 mechanism 1/3)

A 'yes' at any step stops the cascade there: native decoding (strongest guarantee) or a forced tool call (Part 3.2/3.3), instead of falling further.

Output parsers

For cases not using .with_structured_output(), LangChain's output parser classes (StrOutputParser, PydanticOutputParser, JsonOutputParser) handle the "generate freely, then parse/validate" pattern from Part 3.2's mechanism 3 — PydanticOutputParser specifically can also inject formatting instructions into your prompt (describing the expected schema in natural language for the model to follow) and then validate the raw output against your Pydantic model, raising a clear error on mismatch, which you can then wire into a retry pattern exactly as in Part 3.2's production example.

5. Simple mental model

ChatPromptTemplate is a mail-merge template: the template itself (with named blanks) is a stable, versioned document, and each call fills in specific values without touching the template's structure — exactly like a form letter where the letterhead and structure stay fixed while the recipient's name and details change per copy. .with_structured_output() is a universal translator for "please answer in this exact shape," that picks whichever of the underlying enforcement strategies (native constrained decoding, tool-call forcing, or generate-then-validate) actually works for the specific provider you're talking to, so you don't need to know or care which one is happening for a given model.

6. Real-world example

A team supports both Anthropic and OpenAI models behind their extraction pipeline (Part 3.11's fallback pattern). Writing the extraction logic once against model.with_structured_output(InvoiceExtraction) means the same code path works for both providers, even though the underlying enforcement mechanism differs between them — without this abstraction, they'd need separate code paths handling each provider's specific structured-output API shape.

7. Architecture diagram

ChatPromptTemplateversioned, reusable

list[BaseMessage]

model.with_structured_output(YourModel)picks strategy per provider · Part 4.3

Your validated Pydantic instance

8. Production considerations

• Verify which underlying structured-output strategy .with_structured_output() uses for your specific provider/model and confirm it gives you the guarantee strength (Part 3.2) your use case actually requires — don't assume uniform strong guarantees across every provider just because the calling code looks identical.
• Still add your own independent semantic validation (Part 3.2, section 8) even when using .with_structured_output() — the same two-layer discipline applies: LangChain's structured-output abstraction addresses schema conformance, not semantic correctness.
• Version ChatPromptTemplate definitions in your codebase, in Git (Part 1.7), with the same review/eval discipline as any other prompt change (Part 3.1, Part 1.9).

9. Common mistakes

• Assuming .with_structured_output() provides the same strength of guarantee across every provider/model combination without verifying.
• Constructing prompts via ad hoc string formatting scattered through the codebase instead of consolidating them into versioned ChatPromptTemplate objects, losing the reuse/testability benefit this abstraction exists to provide.
• Not adding independent Pydantic validation on top of .with_structured_output()'s result, missing semantic errors the schema-conformance layer can't catch (Part 3.2).

10. Security considerations

• Templates that incorporate untrusted content (retrieved documents, Part 3.5; user input) into a placeholder still carry every prompt-injection consideration from Part 3.1/9.1 — using a ChatPromptTemplate object instead of raw string formatting doesn't add any implicit sanitization of what gets filled into it.

11. Performance considerations

• .with_structured_output()'s underlying strategy affects latency: a native structured-output/constrained-decoding path is typically comparable to a normal generation call, while a generate-then-validate fallback strategy can add retry round trips on validation failure (Part 3.2, section 11) — worth profiling per provider.

12. Cost considerations

• A fallback-strategy retry loop (Part 3.2's mechanism 3) triggered by .with_structured_output() under the hood on a provider without native support adds the same real, measurable retry cost discussed in Part 3.2, section 12 — even though it's hidden behind one convenient method call.

13. When to use it

ChatPromptTemplate for essentially all production prompt construction (the versioning/reuse benefit is close to free); .with_structured_output() whenever you need schema-conformant output and want provider-agnostic code, understanding its underlying strategy for your specific provider.

14. When NOT to use it

If you need to know and control precisely which structured-output enforcement mechanism is used (e.g., you specifically require the strongest constrained-decoding guarantee and must confirm it's actually active), consider using the provider-specific structured-output API directly (as in Part 3.2's raw example) rather than the abstracted .with_structured_output(), so the guarantee is explicit rather than implicit in the abstraction's provider-dependent choice.

15. Alternatives and trade-offs

Approach	Good for	Weak point
ChatPromptTemplate + .with_structured_output()	Provider-agnostic, reusable, less code	Underlying guarantee strength varies by provider, must be verified
Raw provider SDK structured output (Part 3.2)	Explicit, known guarantee strength	Provider-specific code, not portable
Output parser + manual retry (Part 3.2, mechanism 3)	Works with any model, full control over retry logic	More code, weaker guarantee than native constrained decoding

16. Practical Python/code example

from langchain_core.prompts import ChatPromptTemplate
from langchain_anthropic import ChatAnthropic
from pydantic import BaseModel, Field


class InvoiceExtraction(BaseModel):
    """Structured fields extracted from an invoice document."""
    vendor_name: str
    total_amount_usd: float = Field(gt=0)


prompt = ChatPromptTemplate.from_messages([
    ("system", "Extract invoice fields from the provided text."),
    ("user", "{document_text}"),
])
model = ChatAnthropic(model="claude-sonnet-4-5").with_structured_output(InvoiceExtraction)

chain = prompt | model
result: InvoiceExtraction = await chain.ainvoke({"document_text": "Invoice from Acme Corp, total $150.00"})

17. Production-quality example

Adding independent semantic validation on top of the chain, exactly per the section 9 warning:

import logging
from pydantic import ValidationError

logger = logging.getLogger("extraction_chain")


async def extract_with_validation(chain, document_text: str) -> InvoiceExtraction:
    """
    Runs the extraction chain and re-validates the result independently, since
    with_structured_output's schema conformance doesn't guarantee semantic correctness.

    Args:
        chain: The composed prompt | structured-output-model chain.
        document_text (str): The document text to extract from.

    Returns:
        InvoiceExtraction: The validated extraction result.

    Raises:
        ValidationError: If the result fails independent semantic validation.
    """
    result = await chain.ainvoke({"document_text": document_text})
    try:
        return InvoiceExtraction.model_validate(result.model_dump())
    except ValidationError:
        logger.exception("extraction result failed independent semantic validation")
        raise

18. Short exercise

Check current LangChain documentation for whether .with_structured_output() exposes a way to explicitly request a specific underlying strategy (e.g., forcing tool-calling-based extraction rather than a provider's native structured-output feature) for a given model. Note what you found.

19. Interview questions

• What does .with_structured_output() actually do differently depending on the underlying provider, and why does that matter for the guarantee strength you can rely on?
• Why should you still validate structured output independently even when using LangChain's abstraction?
• Why does keeping ChatPromptTemplate definitions separate from runtime values matter for the version-control discipline from Part 1.7?

20. FDE/customer scenario

Customer's engineer: "We switched our extraction pipeline from Anthropic to a different provider and now we're seeing more validation failures — did LangChain break?"

The likely, correct diagnosis (informed by section 4/8): the new provider/model combination is using a weaker underlying structured-output strategy than the previous one (e.g., falling back to generate-then-validate instead of native constrained decoding), which .with_structured_output()'s abstraction doesn't surface loudly — walking through which strategy is actually active for the new provider, rather than assuming a framework bug, is the credible, correct diagnostic path.

---

Key takeaways

• ChatPromptTemplate operationalizes Part 3.1's prompt-versioning discipline as a first-class, composable Runnable object.
• .with_structured_output() abstracts over Part 3.2's different structured-output mechanisms — convenient, but the underlying guarantee strength still varies by provider and must be verified, not assumed.
• Independent semantic validation remains necessary regardless of which structured-output path is used underneath.

Things you should be able to explain

• The three underlying strategies .with_structured_output() might use depending on provider.
• Why the abstraction's convenience doesn't remove the need to verify guarantee strength.

Things you should be able to build

• A versioned prompt template composed with structured output and independent semantic validation.

Common mistakes

• Assuming uniform structured-output guarantee strength across all providers.
• Skipping independent validation because the framework "already validated" the schema.

Recommended next chapter

04-tools.md