3.12 — AI Application Architecture (Synthesis)

1. What is it?

This chapter is a synthesis, not a new technique: it's where every piece from Parts 3.1–3.11 (prompting, structured output, tool calling, vector databases, RAG, hybrid search, agents/workflows, MCP, memory, multimodal, model routing) gets assembled into the shape of a real, complete AI application — and, just as importantly, this chapter is about the decision process for choosing which pieces a given application actually needs, since a real system almost never needs all of them.

2. Why does it exist?

Every previous chapter in Part 3 taught one component in isolation, the way you'd study one instrument before playing in an orchestra. But a real AI application is never "just RAG" or "just an agent" — it's a composition of several of these pieces, wired together to serve a specific business need, with specific latency/cost/reliability requirements. Without a chapter that steps back and shows how to compose these pieces deliberately, there's a real risk of learning each technique well individually while still not knowing how to architect a coherent, appropriately-scoped real system — which is precisely the skill an AI FDE is judged on.

3. What problem does it solve?

It solves "given a real business requirement, which of these dozen techniques do I actually need, in what combination, and in what order do I make these architectural decisions" — the concrete design process, not just the component catalog. This chapter also explicitly sets up Part 11 (System Design for AI), which will apply this same decision process to several full, complex enterprise system designs end-to-end.

4. How does it work internally?

The decision tree that ties Part 3 together

When approaching a new AI application requirement, the components from this Part apply in roughly this decision order (though real design work often iterates rather than proceeding perfectly linearly):

1. Does this need an LLM at all, or does a deterministic system solve it better?
   (Part 2.1/2.3: classical ML/NLP or plain code, if the task is narrow/deterministic)
        │ needs LLM judgment/generation
        ▼
2. Does the model need facts/data it doesn't have?
   ├── Precise, structured, exact-match data → Tool calling / SQL (Part 3.3, 1.4)
   ├── Large, unstructured, semantically-searchable knowledge → RAG (Part 3.5)
   │      └── Retrieval quality gaps? → hybrid search / reranking / query
   │          rewriting (Part 3.6), only where measured to be needed
   └── No external data needed → prompting alone may suffice (Part 3.1)
        │
        ▼
3. Does the model's output feed into code/another system?
   → Structured output enforcement (Part 3.2), always, for any code-consumed output
        │
        ▼
4. Is the sequence of steps to accomplish the task known in advance?
   ├── Yes → build a workflow (Part 3.7) — fixed control flow, LLM judgment
   │          only at specific points
   └── No, genuinely unpredictable → an agent is justified (Part 3.7), bounded
        │
        ▼
5. Will this tool/data integration be reused across multiple AI applications?
   → Consider MCP (Part 3.8) for the integration layer
        │
        ▼
6. Does the interaction span multiple turns or sessions?
   → Memory strategy (Part 3.9): short-term context management,
     plus long-term structured/semantic memory if cross-session recall matters
        │
        ▼
7. Does meaningfully relevant information exist in non-text form?
   → Multimodal input handling (Part 3.10), with a hybrid fallback to
     specialized models for precision-critical sub-tasks
        │
        ▼
8. Given the mix of task types now assembled, does complexity vary enough
   to benefit from routing?
   → Model routing (Part 3.11) across the different components/steps

This is not a checklist where every box must be checked — it's a decision filter, and for most real applications, several steps will correctly resolve to "not needed for this system." A well-architected system is often notable for what it deliberately excludes as much as for what it includes.

The layered request-flow view

Assembling several of these components into one coherent request flow (using a customer support assistant as the running example threaded through this Part):

1. Incoming user message

2. Memoryshort-term context + long-term facts · Part 3.9

3. Model routingPart 3.11

4. Workflow routingclassify intent · Part 3.7

5a. RAG pathhybrid retrieve + rerank + citations

5b. Tool-call pathbounded loop, maybe via MCP

6. Final response generationstructured or natural language

7. Update memoryfacts + conversation history

Each numbered stage maps directly to a specific chapter in this Part.

Notice each numbered stage maps directly to a specific chapter in this Part — this is the concrete, assembled shape that the individual techniques were building toward all along.

5. Simple mental model

If Parts 3.1–3.11 each taught you one musical instrument, this chapter is learning to read a score and conduct an orchestra — knowing which instruments a given piece of music actually calls for (not every piece needs percussion, brass, and strings all at once), in what order they enter, and how they combine into something coherent rather than a wall of every instrument playing at maximum volume simultaneously. An overengineered AI system — RAG plus agents plus MCP plus elaborate memory, deployed for a task that needed only a single well-prompted classification call — is the engineering equivalent of a string quartet's simple piece scored for a full orchestra: technically more "impressive" sounding component list, actually worse for the actual piece.

6. Real-world example

An early design for an internal IT helpdesk assistant proposed: a full RAG pipeline over IT documentation, a fully autonomous multi-step agent for troubleshooting, long-term semantic memory of every past interaction, and multimodal support for screenshot uploads — essentially every technique in this Part, all at once, from day one. Applying the decision tree from section 4 to the actual, measured request distribution revealed: 60% of requests were simple, well-defined lookups ("how do I reset my VPN") answerable by a lightweight RAG pipeline with no agent needed at all; 30% needed a bounded, fixed workflow (check ticket status → escalate if unresolved) rather than an open-ended agent; only about 10% of genuinely novel troubleshooting cases arguably justified agent-level flexibility. Redesigning around this actual distribution — mostly RAG and workflows, a narrow, tightly-bounded agent only for the genuinely unpredictable 10% — produced a system that was simpler to build, cheaper to run, far more predictable to debug, and (measured via evaluation, Part 8) no less effective for users, since the components that were cut weren't actually adding measured value for the majority of real traffic.

7. Architecture diagram

See section 4's layered request-flow diagram — this chapter's architecture is that diagram, generalized as the composition pattern for real AI applications built from this Part's components.

8. Production considerations

• Build incrementally, adding a component only when a measured gap justifies it — start with the simplest architecture that could plausibly work (often: prompting alone, or prompting plus one RAG pipeline), measure its actual failure modes against real usage and evaluation (Part 8), and add exactly the component that addresses the measured gap, not a component that sounds sophisticated.
• Each added component is an added operational surface — more moving pieces to monitor (Part 8.4), more places for something to fail (Part 7), more attack surface (Part 9) — the cost of complexity is real and compounds, and should be weighed explicitly against the measured benefit each addition provides.
• Document the actual architecture, including what was deliberately excluded and why — a design record noting "we considered an agent architecture here but chose a fixed workflow because the task's steps are fully known" is valuable both for future maintainers and for explaining the design credibly to a customer's technical stakeholders.

9. Common mistakes

• Architecting for the most impressive-sounding, most complete component list rather than for the actual, measured requirements of the specific application — a very common failure mode this entire Part has been building the judgment to avoid.
• Adding components speculatively ("we might need agents eventually") rather than incrementally, in response to a measured, current gap.
• Not revisiting architecture decisions as real usage data accumulates — an architecture appropriate at launch, based on assumptions about usage patterns, may need to shift once real, measured usage reveals different actual needs (Part 8's evaluation and monitoring discipline is what makes this revision possible and evidence-based rather than guesswork).

10. Security considerations

• Every component added multiplies the security surface covered across Parts 3.1–3.11's individual security sections — a system combining RAG, tool calling, and agents inherits the injection risks of RAG (Part 9.1) and the excessive-agency risks of tool calling/agents (Part 9.2) simultaneously, and these risks can interact (an indirect injection via a retrieved document influencing an agent's subsequent tool-calling decisions) in ways that are harder to reason about than either risk in isolation — a direct, concrete argument for minimizing architectural complexity to only what's actually needed.

11. Performance considerations

• Latency is additive across the pipeline stages in section 4's diagram — a system that adds memory retrieval, model routing, hybrid search, reranking, and a multi-step agent all in sequence for every single request will have meaningfully higher end-to-end latency than a system that includes only the stages a given request actually needs (a well-designed routing/workflow layer, Part 3.7, is precisely what lets simple requests skip unnecessary stages rather than paying every stage's cost regardless of need).

12. Cost considerations

• Cost, like latency, is additive across included components — every RAG retrieval call, every reranking pass, every extra agent iteration, every model call in a multi-step workflow adds to the per-request cost (Part 3.11, Part 7.10) — the discipline of including only measured-necessary components is as much a cost-control practice as it is a reliability one.

13. When to use this decision process

For every new AI application or feature, before writing implementation code — treat the decision tree in section 4 as a design exercise to work through explicitly (even briefly, on paper or in a design doc) rather than defaulting to whatever combination of techniques feels most current or impressive.

14. When NOT to over-apply this process

Don't treat the decision tree as requiring a lengthy formal exercise for genuinely trivial features (a single, simple, well-understood LLM call with no external data needs) — the process should scale its own rigor to the actual complexity and stakes of what's being built, which is itself an application of the same "don't over-engineer" principle this chapter argues for.

15. Alternatives and trade-offs

This chapter's "alternative" is really a meta-point: the alternative to deliberate, incremental, evidence-based architecture is either under-building (a naive single-LLM-call system that fails on real requirements it didn't anticipate) or over-building (a maximal, all-components system that's expensive, slow, and hard to debug relative to what the actual requirements justify) — both are real, common failure modes, and the decision process in this chapter exists specifically to avoid both.

16. Practical Python/code example

A minimal illustration of the routing-first architecture from section 4 — the top-level dispatcher that decides which downstream pipeline (from earlier chapters) a given request actually needs:

from enum import Enum


class RequestPath(str, Enum):
    """The set of downstream pipelines a request can be routed to."""
    DIRECT_RESPONSE = "direct_response"
    RAG = "rag"
    TOOL_CALL = "tool_call"
    AGENT = "agent"


async def classify_request_path(client, user_message: str) -> RequestPath:
    """
    Classifies which downstream pipeline a request needs, so simple requests can
    skip the cost/latency of pipelines they don't require.

    Args:
        client: An async LLM client.
        user_message (str): The incoming user message.

    Returns:
        RequestPath: The pipeline this request should be routed to.
    """
    response = await client.messages.create(
        model="claude-haiku-4-5",  # a small, fast model is appropriate for this classification step
        max_tokens=10,
        system=(
            f"Classify the request into exactly one of: {[p.value for p in RequestPath]}. "
            "direct_response: general conversation needing no external data. "
            "rag: needs knowledge-base/document lookup. "
            "tool_call: needs a specific, single real-time data lookup or action. "
            "agent: needs open-ended, multi-step investigation. "
            "Respond with only the category."
        ),
        messages=[{"role": "user", "content": user_message}],
    )
    return RequestPath(response.content[0].text.strip().lower())


async def handle_request(client, user_message: str, tenant_id: str) -> str:
    """
    Dispatches a request to the appropriate pipeline based on its classified path,
    ensuring each request only incurs the cost/latency of the components it needs.

    Args:
        client: An async LLM client.
        user_message (str): The incoming user message.
        tenant_id (str): Tenant scope for any retrieval/tool operations.

    Returns:
        str: The final response text.
    """
    path = await classify_request_path(client, user_message)

    if path == RequestPath.DIRECT_RESPONSE:
        response = await client.messages.create(
            model="claude-haiku-4-5", max_tokens=500, messages=[{"role": "user", "content": user_message}]
        )
        return response.content[0].text
    elif path == RequestPath.RAG:
        result = await answer_with_rag(client, vector_store, user_message, tenant_id)
        return result["answer"]
    elif path == RequestPath.TOOL_CALL:
        return await handle_support_message(user_message, tenant_id)
    else:
        trace = await run_bounded_agent(client, TOOLS, TOOL_IMPLEMENTATIONS, user_message)
        return trace.final_answer or "I wasn't able to complete this request."

This single dispatcher is the concrete embodiment of section 4's decision tree — every request pays only for the pipeline stages its own classified path actually requires, directly avoiding the over-building failure mode from section 15.

17. Production-quality example

A design-record template, making the "document what was excluded and why" production consideration from section 8 concrete and reusable across projects:

# AI Architecture Decision Record: [Feature Name]

## Requirement
[One paragraph: the actual business need, in plain language]

## Components included, and why
- [Component, e.g., "RAG over product documentation"]: [why it's needed —
  what measured or clearly anticipated gap it closes]

## Components explicitly excluded, and why
- [Component, e.g., "Open-ended agent"]: [why it was NOT included — e.g.,
  "the task's steps are fully known in advance; a fixed workflow is more
  predictable and auditable for this specific process"]

## Revisit triggers
[What future evidence — usage data, evaluation results, a new requirement —
would justify reconsidering an excluded component or adding a new one]

Using this template on every new feature turns "we thought about complexity and decided against it" from an informal, easily-forgotten judgment call into a documented, revisitable decision — directly useful both for future team members and for explaining architecture choices credibly to a customer's technical reviewers (Part 12).

18. Short exercise

A customer describes a desired feature: "Let employees ask our AI assistant questions about our HR policies, and let it also submit a time-off request on their behalf when asked." Walk through the decision tree in section 4 explicitly for this feature, and produce a short architecture decision record (using the section 17 template) stating which components you'd include, which you'd exclude, and why.

19. Interview questions

• Walk through your decision process for determining whether a given AI feature needs RAG, tool calling, an agent, or some combination — using a concrete example.
• Describe a real or hypothetical case where you'd deliberately choose NOT to use an agent, RAG, or memory, even though the feature could technically benefit from it, and justify the trade-off.
• Why does architectural complexity compound cost, latency, and security risk simultaneously, and what does that imply about how aggressively to add components?

20. FDE/customer scenario

Customer: "We've seen impressive demos with agents and RAG and memory all together — shouldn't our system have all of that too, to be competitive?"

This is the single most direct, practically important conversation this chapter prepares an FDE to have. The credible, trust-building response isn't "yes, let's build all of it" — it's walking the customer through the same decision process as section 4, applied to their actual requirements, and showing (often to their genuine relief, once cost and timeline trade-offs are made concrete) that a simpler, more targeted architecture will serve their real use case better, faster, and cheaper than matching every component seen in someone else's demo. This is precisely the FDE-differentiating judgment that prompt.md's teaching rule ("never teach a technology in isolation... connect business problem through to business impact") has been building toward across this entire Part.

---

Key takeaways

• Real AI applications are compositions of the individual techniques from Parts 3.1–3.11, chosen deliberately per actual requirement, not a fixed checklist to fully satisfy.
• The right default is the simplest architecture that could plausibly work, built incrementally in response to measured gaps — not the most complete-sounding component list.
• Every added architectural component compounds latency, cost, and security surface — complexity has a real, cumulative price that must be justified by measured benefit.

Things you should be able to explain

• The decision tree for choosing which Part 3 components a given application actually needs.
• Why over-building (maximal component inclusion) is as real a failure mode as under-building.

Things you should be able to build

• A top-level request router that dispatches to only the pipeline stages a given request actually needs.
• An architecture decision record documenting both included and deliberately excluded components.

Common mistakes

• Architecting for an impressive component list instead of measured requirements.
• Adding components speculatively rather than in response to evidence.
• Never revisiting architecture as real usage data accumulates.

Recommended next chapter

Part 3 complete. Continue to handbook/04-langchain/01-architecture-internals.md.