4.6 — Middleware, Streaming, Async, and Callbacks

1. What is it?

This chapter covers four related LangChain mechanisms for observing and controlling execution: the middleware system (introduced in LangChain 1.0, Part 4.1) for hooking into an agent's loop; streaming (.stream()/.astream()) for incremental output; async (.ainvoke(), .abatch(), .astream()) as the concurrency model (Part 1.1) applied throughout the framework; and callbacks, the older, more general-purpose event-hook system for observing execution at a granular level (model calls, chain steps, tool calls) — which is also the mechanism LangSmith's tracing (Part 6.1) is built on.

2. Why does it exist?

Part 3.7's agent loop and Part 1.1's async discipline both established real production needs: bounding/observing an agent's execution, and not blocking on slow I/O. Streaming exists because Part 2.6/2.4 established that generation is inherently sequential — showing a user tokens as they're generated, rather than waiting for the complete response, is a genuine, measurable perceived-latency improvement (Part 7 will formalize "time to first token" as a metric this directly serves). Callbacks exist as a general instrumentation point predating the 1.0 middleware system, giving fine-grained visibility into every step of execution — which is precisely the mechanism that makes LangSmith's automatic tracing (Part 6.1) possible without you writing manual logging into every component.

3. What problem does it solve?

Middleware solves "how do I add cross-cutting production concerns (retry, summarization, PII redaction, human-in-the-loop) to an agent loop declaratively" (Part 4.1). Streaming solves "how do I reduce perceived latency for a user waiting on a long generation." Async solves "how do I handle concurrent requests efficiently" (Part 1.1, applied through the framework's Runnable interface). Callbacks solve "how do I observe/instrument what's happening inside a chain or agent without modifying its core logic" — the foundation for both debugging and production observability (Part 6, Part 8.4).

4. How does it work internally?

Middleware — hook points in the agent loop

Recall Part 4.1's description: middleware are functions/classes that run at defined points in create_agent's loop — before the model is called, after the model responds, before a tool executes. Mechanically, this means the agent's loop (structurally identical to Part 3.7's bounded tool-calling loop) has explicit extension points where registered middleware can inspect, modify, or short-circuit the flow:

before_model hookscan modify the request or short-circuit

Model is called

after_model hookscan modify the response, e.g. summarize

before_tool_execution hookse.g. human-in-the-loop approval

Tool executes

A SummarizationMiddleware (Part 4.1) hooks into before_model, checking accumulated conversation length and replacing older messages with a summary exactly as Part 3.9's manual example did — but as a reusable, declaratively-attached component rather than hand-written logic threaded through your own loop code. A human-in-the-loop middleware hooks into before_tool_execution, pausing the loop (using LangGraph's interrupt mechanism underneath, Part 5.4, since create_agent is built on LangGraph, Part 4.1) until an external approval signal resumes it.

Streaming — what's actually being streamed, and streaming modes

.stream()/.astream() on a Runnable yields incremental chunks as they become available rather than waiting for the full result. For a chat model, this means individual tokens (or small token groups) as the provider generates them — implemented by keeping the underlying HTTP connection open and reading the provider's streamed response incrementally (typically Server-Sent Events under the hood) rather than waiting for a single complete HTTP response.

For an agent or multi-step chain built with LangGraph underneath (Part 5.5 covers this in full depth), streaming has multiple distinct modes worth distinguishing: streaming the final output's tokens as they're generated is different from streaming state updates as the agent progresses through multiple steps (e.g., "now calling tool X," "now generating final response") — the latter is what lets a user-facing UI show "the agent is looking up your order..." rather than a single opaque loading spinner for a multi-step agent run. Verify which streaming mode(s) a given LangChain/LangGraph version and component actually support before designing a UI around an assumption about what granularity of streaming is available.

Async throughout — inheriting Part 1.1's discipline

Every Runnable's .ainvoke()/.abatch()/.astream() methods are the async counterparts to their sync equivalents, and Part 1.1's entire discussion (don't block the event loop, use async clients throughout, bound concurrency) applies directly: a LangChain chain built entirely from async-compatible components genuinely benefits from Part 1.1's concurrency model, but mixing in a synchronous component (a sync-only tool implementation, a blocking database call inside a tool) reproduces exactly the event-loop-freezing failure mode Part 1.1 warned about, now hidden one layer deeper inside a framework abstraction where it may be less obvious to spot.

Callbacks — the general instrumentation layer

from langchain_core.callbacks import BaseCallbackHandler

class LoggingCallback(BaseCallbackHandler):
    async def on_llm_start(self, serialized, prompts, **kwargs):
        logger.info("LLM call starting: %s", prompts)

    async def on_tool_end(self, output, **kwargs):
        logger.info("Tool finished: %s", output)

Callback handlers implement methods corresponding to specific lifecycle events (on_llm_start, on_llm_end, on_tool_start, on_tool_end, on_chain_start, on_chain_end, and more) and are attached to a run via a callbacks=[...] parameter or a run-scoped configuration. Every component in the framework — chains, models, tools, retrievers — invokes these callback methods at the appropriate points in its execution, regardless of what the component actually does internally, which is precisely what makes callbacks a uniform instrumentation layer across the entire framework rather than something you'd need to hand-wire per component. This is the actual mechanism LangSmith's automatic tracing (Part 6.1) plugs into: LangSmith registers a callback handler that records every one of these events into a structured trace, without your application code needing to call any LangSmith-specific logging itself.

5. Simple mental model

Middleware is like airport security checkpoints along a fixed route — the traveler (the agent loop) follows a defined path, and checkpoints (middleware) at specific points can inspect, modify (repack a bag), or halt (require additional screening) the traveler's progress, without the traveler's own itinerary needing to know about the checkpoints in advance. Callbacks are like security cameras positioned throughout the whole building — passively recording everything that happens everywhere, for later review (LangSmith's tracing), without altering the flow of anything, unlike middleware which can actively intervene.

6. Real-world example

A team building a customer support agent needs three things simultaneously: streaming the response so the UI shows tokens as they arrive (better perceived latency), a middleware-based human-approval step before any tool that modifies a customer's account (a real safety requirement), and full tracing of every step for later debugging (via LangSmith's callback-based integration, Part 6.1) — all three mechanisms from this chapter working together on the same underlying agent, each addressing a genuinely distinct concern (perceived latency, safety control, observability) rather than any one of them substituting for the others.

7. Architecture diagram

Agent loop · Part 3.7/4.1

Callbacksfire at every lifecycle event — passive recording, e.g. LangSmith tracing (Part 6.1)

Middleware hooksactive intervention at defined points — summarization, PII redaction, human-in-the-loop approval

Model callcan stream tokens incrementally (.astream())

Tool callmiddleware can pause for approval before executing

Callbacks observe passively; middleware can actively intervene at either step.

8. Production considerations

• Attach a tracing callback handler (or LangSmith integration, Part 6.1) to every production agent/chain by default — the passive, uniform instrumentation callbacks provide is close to a prerequisite for debugging real production incidents (Part 8.4), and it's far more expensive to add retroactively after an incident than to have running from day one.
• Design human-in-the-loop middleware checkpoints deliberately around genuinely high-stakes actions (Part 3.7, Part 5.4, Part 8.5) — not every tool call needs a pause, and over-using approval checkpoints defeats the purpose of automation while under-using them misses the safety benefit for the actions that actually need it.
• Verify streaming mode support for your specific component combination before committing a UI design to a particular granularity of incremental updates.
• Audit every custom tool/component in an async chain for accidental blocking calls (Part 1.1) — this failure mode is easier to miss when it's nested inside a framework-orchestrated chain than in code you wrote and can see end-to-end directly.

9. Common mistakes

• Building a production agent with no tracing/callback instrumentation attached, then having no way to reconstruct what happened during a specific customer-reported incident after the fact.
• Adding human-in-the-loop middleware to every tool call indiscriculately, creating a system so gated it defeats the purpose of automation, versus reserving it for genuinely high-stakes actions (Part 3.7's loan-review example distinction).
• Mixing a synchronous tool implementation into an otherwise-async agent, reintroducing Part 1.1's event-loop-blocking failure mode one layer removed from visibility.
• Assuming streaming automatically means lower total latency — streaming improves perceived latency (time to first visible content) but doesn't necessarily reduce the total time for a full response to complete (Part 2.6's tokens-per-second discussion).

10. Security considerations

• Callback handlers that log full request/response content (common for debugging) must handle PII/sensitive data with the same care as any other logging (Part 9.6) — a verbose tracing callback is itself a place where sensitive conversation content can end up over-retained or under-protected if not deliberately scoped.
• Human-in-the-loop middleware is a genuine security control for excessive-agency risk (Part 9.2) — but only as strong as the actual review process behind the "approval" signal; a rubber-stamp approval process provides much less real safety than the architecture diagram suggests.

11. Performance considerations

• Streaming's main benefit is perceived latency (time to first token), not total throughput — measure and report both metrics separately (Part 2.6, Part 7) rather than conflating "feels faster" with "is faster."
• Excessive middleware/callback overhead (very heavy processing in a hook that fires on every single step) can add real, measurable latency to a high-frequency agent loop — keep hook logic lightweight, especially for hooks that fire on every iteration.

12. Cost considerations

• Summarization middleware trades a periodic extra LLM call for bounded ongoing context size (Part 3.9's trade-off, now as a reusable component) — the cost trade-off analysis from Part 3.9 applies identically.
• Tracing/callback overhead itself has no direct LLM-token cost, but a hosted tracing service (LangSmith, Part 6) typically has its own usage-based pricing worth accounting for at scale (Part 6.1 covers this).

13. When to use it

Tracing/callbacks: essentially always, for any production system (the observability benefit is close to unconditionally worth the modest setup cost). Streaming: any user-facing interactive application where perceived latency matters. Middleware: whenever your agent needs one of its specific cross-cutting concerns (context management, safety gating, redaction) rather than hand-writing that logic into your own loop.

14. When NOT to use it

Streaming may not be worth the added complexity for backend, non-interactive batch processing where nothing is waiting on incremental output. Middleware for a workflow (Part 3.7) rather than an agent may be unnecessary — a fixed-sequence workflow's cross-cutting concerns (logging, redaction) can often be implemented as plain function calls at each fixed step, without needing the dynamic hook-based middleware system designed for open-ended agent loops.

15. Alternatives and trade-offs

Mechanism	Good for	Weak point
Middleware	Declarative, reusable cross-cutting agent-loop concerns	Only applies to agent-loop-shaped execution (create_agent)
Callbacks	Passive, uniform observability across any component	Doesn't intervene/modify execution, only observes
Streaming	Better perceived latency for interactive use cases	Doesn't reduce total latency/cost; adds some implementation complexity
Hand-written logic (no framework mechanism)	Full control, no framework dependency	Reimplements what these mechanisms already provide, more maintenance

16. Practical Python/code example

from langchain_core.callbacks import BaseCallbackHandler
import logging

logger = logging.getLogger("agent_trace")


class ProductionLoggingCallback(BaseCallbackHandler):
    """Logs key lifecycle events for production debugging, independent of any
    hosted tracing service."""

    async def on_llm_start(self, serialized, prompts, **kwargs):
        logger.info("LLM call starting")

    async def on_tool_start(self, serialized, input_str, **kwargs):
        logger.info("Tool call starting: %s input=%s", serialized.get("name"), input_str)

    async def on_tool_end(self, output, **kwargs):
        logger.info("Tool call finished: %s", output)

    async def on_chain_error(self, error, **kwargs):
        logger.error("Chain execution failed: %s", error)


result = await agent.ainvoke(
    {"messages": [{"role": "user", "content": "Where is my order?"}]},
    config={"callbacks": [ProductionLoggingCallback()]},
)

17. Production-quality example

Streaming an agent's incremental progress to a user-facing interface, distinguishing token-level streaming from step-level status updates (section 4's distinction):

async def stream_agent_progress(agent, user_message: str):
    """
    Streams both step-level status updates and final-response tokens to a
    consuming interface, giving visibility into a multi-step agent run rather
    than a single opaque loading state.

    Args:
        agent: A create_agent-built agent (LangChain, built on LangGraph, Part 4.1).
        user_message (str): The user's message.

    Yields:
        dict: Either a status update ({"type": "status", "detail": ...}) or a
            token chunk ({"type": "token", "text": ...}), for the consuming UI
            to render appropriately.
    """
    async for event in agent.astream_events(
        {"messages": [{"role": "user", "content": user_message}]}, version="v2"
    ):
        # Verify exact current event schema against current LangChain/LangGraph
        # documentation — astream_events' event shape is a part of the API
        # worth double-checking at write time.
        if event["event"] == "on_tool_start":
            yield {"type": "status", "detail": f"Looking up {event['name']}..."}
        elif event["event"] == "on_chat_model_stream":
            chunk = event["data"]["chunk"]
            if chunk.content:
                yield {"type": "token", "text": chunk.content}

18. Short exercise

Design (in plain language, no code required) a middleware you'd want for a production agent that handles financial transactions: what hook point(s) would it use, what would it check before allowing execution to continue, and what would happen if the check failed? Reference Part 3.7's excessive-agency discussion in your justification for where this check belongs.

19. Interview questions

• Explain the difference between middleware and callbacks in LangChain's execution model — what can each do that the other can't?
• Why does streaming improve perceived latency without necessarily improving total latency or cost?
• What's the underlying mechanism that lets LangSmith automatically trace a LangChain application without the application writing LangSmith-specific logging code?

20. FDE/customer scenario

Customer: "When something goes wrong with the assistant, we have no way to see what it actually did step by step."

This is a direct, fixable observability gap: attaching a proper callback-based logging handler (or, more robustly, LangSmith tracing, Part 6.1) retroactively is a relatively fast, high-value fix compared to the cost of continuing to operate a production AI system with no visibility into its own execution — and being able to propose this as a concrete, quickly-actionable improvement (rather than a vague "we should add more logging") is a good example of translating this chapter's mechanism into a credible customer-facing recommendation.

---

Key takeaways

• Middleware actively intervenes in an agent's loop at defined hook points (retry, redaction, human-in-the-loop); callbacks passively observe execution across any component — different tools for different needs.
• Streaming improves perceived latency (time to first token/status), not necessarily total latency or cost.
• Async discipline (Part 1.1) applies throughout the framework — a single blocking component inside an async chain reproduces the same event-loop-freezing failure mode, just harder to spot inside framework abstractions.

Things you should be able to explain

• The difference between middleware and callbacks, and when each is the right tool.
• Why streaming doesn't reduce total generation time.

Things you should be able to build

• A production logging callback handler and a step-aware streaming consumer distinguishing status updates from token output.

Common mistakes

• No tracing/observability instrumentation on production agents.
• Over-applying human-in-the-loop middleware to every action, defeating automation's purpose.
• Hidden blocking calls inside framework-orchestrated async chains.

Recommended next chapter

07-production-patterns.md