11.1 — AI System Design: A Framework From First Principles

1. What is it?

This chapter establishes the methodology used consistently across every system design in this Part (11.2–11.9): a repeatable, first-principles process for going from a vague business need to a fully-reasoned architecture, covering requirements, constraints, components, data flow, failure modes, security, scalability, observability, cost, and trade-offs — in that order, deliberately.

2. Why does it exist?

Part 3.12 taught architecture decisions for a single AI application; this chapter scales that same discipline to full system design — the level of reasoning an AI FDE needs for both real engagements (Part 12/13) and technical interviews (Part 17). Without an explicit, repeatable methodology, system design work risks becoming either an unstructured list of technologies ("we'll use RAG, agents, Kubernetes...") with no clear reasoning connecting them to actual requirements, or a design that looks impressive but silently skips hard questions (failure modes, cost, security) that only surface once it's already in production.

3. What problem does it solve?

It solves "how do I reason through a complex AI system design in a way that's thorough, defensible, and clearly connects every technology choice back to an actual requirement" — both for building real systems and for demonstrating this reasoning process clearly to a customer or an interviewer, since the process of getting to an architecture is often more revealing of genuine understanding than the final architecture diagram alone.

4. How does it work internally? — The 14-step process

Every system design chapter in this Part (11.2–11.9) follows this exact sequence:

1.  Requirements       — what does the customer/business actually need?
2.  Constraints         — what real-world limits shape the solution (budget,
                          timeline, compliance, existing infrastructure)?
3.  Functional requirements — specific, concrete capabilities the system
                          must have
4.  Non-functional requirements — latency, availability, scale, security
                          bars the system must meet
5.  Architecture         — the high-level design, usually as a diagram
6.  Components           — what each major piece does and why it exists
7.  Data flow            — how information actually moves through the system,
                          step by step
8.  Failure modes         — what breaks, and what happens when it does
9.  Security              — the specific threats and mitigations relevant
                          to this system (Part 9)
10. Scalability           — how the system grows to handle more load/data
11. Observability          — how you'd know the system is healthy (Part 8.4)
12. Cost                   — the real, itemized cost drivers (Part 7.10)
13. Trade-offs              — what was given up for what was gained
14. Alternatives             — other viable designs and why this one was chosen

Why this specific order matters

Requirements and constraints come first because every subsequent decision should trace back to one of them — an architecture chosen before requirements are clear is architecture chosen by habit or hype, not by reasoning (exactly Part 3.12's core warning, applied at system scale). Failure modes, security, and cost come before the design is considered "done," not as an afterthought bolted on at the end — a design that hasn't been stress-tested against these three lenses hasn't actually been designed yet, only sketched. Trade-offs and alternatives come last specifically because they can only be meaningfully articulated once the actual chosen design is fully worked out — "what did we give up" only makes sense in contrast to a concrete "what did we choose."

Functional vs. non-functional requirements — a distinction worth being precise about

• Functional: what the system does — "users can ask questions about our product documentation and get an accurate answer with a citation."
• Non-functional: how well the system does it — "95% of responses return within 3 seconds," "the system supports 10,000 concurrent users," "no cross-tenant data leakage under any circumstance" (Part 9.6).

Both categories matter equally, but functional requirements tend to dominate initial customer conversations (Part 12) while non-functional requirements are frequently under-specified until they cause a production problem — an AI FDE's job includes actively surfacing non-functional requirements a customer hasn't thought to state explicitly (Part 12 formalizes this discovery discipline).

5. Simple mental model

This 14-step process is like an architect's process for designing a building, not just sketching what it should look like: understand what the building needs to do and for whom (requirements), understand the site's real constraints (soil conditions, budget, zoning laws), design the structure, then — critically, before calling the design finished — stress-test it against failure scenarios (what happens in an earthquake, a fire, a power outage), security (who can get in, and to where), and cost, before finally being able to articulate clearly what was traded off in reaching this specific design versus other viable ones. Skipping straight to "here's a nice-looking building sketch" without this full process is how you get buildings (or systems) that look fine until the first real stress test.

6. Real-world example

A customer's initial request — "we want an AI assistant for our internal help desk" — contains almost no explicit non-functional requirements. Working through this chapter's process surfaces critical, unstated requirements: does it need to work for employees across multiple countries (data residency, Part 7.3/10.5)? What's the expected concurrent user count during a company-wide outage when everyone hits the help desk at once (scalability, Part 7.5)? What happens if the LLM provider has an outage during that same high-traffic moment (failure modes, Part 7.9)? None of these were in the customer's initial one-sentence request, but all of them materially shape the correct architecture — exactly why steps 1-4 of this process exist as explicit, deliberate steps rather than assumed to be obvious.

7. Architecture diagram

This chapter's "diagram" is the 14-step process itself (section 4) — every subsequent system design chapter (11.2–11.9) produces an actual architecture diagram as step 5 of applying this process to a specific business problem.

8. Production considerations

• Always start from requirements and constraints, never from a technology choice — "we'll use agents" decided before requirements are understood is backward reasoning, precisely the anti-pattern Part 3.12 warned against, now at system-design scale.
• Treat failure modes, security, and cost as integral design steps, not a post-design checklist — a design that hasn't accounted for these isn't finished, even if it looks complete on a diagram.
• Actively surface unstated non-functional requirements (section 6) rather than waiting for a customer to think to mention them — this is a core, high-value FDE skill formalized further in Part 12.

9. Common mistakes

• Jumping straight to architecture/technology choices before requirements and constraints are actually understood.
• Treating non-functional requirements as an afterthought, only discovering critical ones (data residency, concurrent-load expectations, failure-mode tolerance) after the system is already built around assumptions that don't hold.
• Presenting a design without having stress-tested it against failure modes, security, and cost — three lenses that reliably surface real, otherwise-invisible problems.
• Skipping the alternatives/trade-offs discussion, presenting the chosen design as the only reasonable option rather than demonstrating the reasoning that led to choosing it over real alternatives.

10. Security considerations

Security is one of the 14 explicit steps in this process (step 9), never an afterthought — every system design in this Part will apply Part 9's specific attack categories (prompt injection, excessive agency, data exfiltration, tenant isolation) as a deliberate, dedicated step, not a passing mention.

11. Performance considerations

Non-functional requirements (step 4) are where performance targets are made explicit and concrete — a design without stated latency/throughput targets has no way to verify whether it actually succeeded at meeting real performance needs.

12. Cost considerations

Cost (step 12) is an explicit, dedicated step using Part 7.10's cost framework — never estimated vaguely or omitted, since a design's viability is frequently determined as much by its cost profile as by its technical correctness (Part 15's ROI framing depends on this being done rigorously).

13. When to use it

Every non-trivial AI system design task — real customer engagements (Part 12/13), technical interviews (Part 17), and internal architecture proposals — benefits from this explicit, repeatable structure rather than ad hoc reasoning.

14. When NOT to over-apply it

A genuinely small, simple feature addition to an already-well-understood system doesn't need the full 14-step formal treatment — but even here, briefly confirming requirements and considering failure modes/security/cost remains worthwhile, scaled down proportionally (Part 3.12's proportionality principle) rather than skipped entirely.

15. Alternatives and trade-offs

This chapter's own "alternative" is the choice between structured, explicit system-design reasoning (this 14-step process) versus ad hoc, intuition-driven design — the former reliably produces more defensible, complete designs and, critically, is far more legible to a customer or interviewer evaluating your reasoning, not just your final answer.

16. Practical Python/code example

A requirements-and-constraints intake template, operationalizing steps 1-4 as a concrete artifact to fill out before any architecture work begins:

from dataclasses import dataclass, field


@dataclass
class SystemDesignBrief:
    """A structured intake capturing steps 1-4 of the system design process,
    completed BEFORE any architecture decisions are made."""
    business_requirement: str
    constraints: list[str] = field(default_factory=list)  # budget, timeline, compliance, existing infra
    functional_requirements: list[str] = field(default_factory=list)
    non_functional_requirements: dict[str, str] = field(default_factory=dict)
    # e.g., {"latency_p95": "3 seconds", "concurrent_users": "10,000",
    #        "data_residency": "EU only", "availability": "99.9%"}

    def is_ready_for_architecture(self) -> bool:
        """Checks whether enough has been captured to begin architecture design responsibly."""
        return bool(
            self.functional_requirements
            and self.non_functional_requirements
            and self.constraints
        )

17. Production-quality example

Given this chapter is methodology rather than implementation, its "production-quality example" is the discipline of documenting the full 14-step reasoning as a design record — directly extending Part 3.12's architecture decision record template to full system scale:

# System Design Record: [System Name]

## 1. Requirements
[Business need in plain language]

## 2. Constraints
[Budget, timeline, compliance, existing infrastructure]

## 3. Functional Requirements
[Specific, concrete capabilities]

## 4. Non-Functional Requirements
[Latency, availability, scale, security bars — MADE EXPLICIT, not assumed]

## 5. Architecture
[Diagram]

## 6. Components
[What each piece does and why]

## 7. Data Flow
[Step-by-step, referencing the diagram]

## 8. Failure Modes
[What breaks, what happens]

## 9. Security
[Specific threats (Part 9) and mitigations]

## 10. Scalability
[How the system grows]

## 11. Observability
[How you'd know it's healthy, Part 8.4]

## 12. Cost
[Itemized cost drivers, Part 7.10]

## 13. Trade-offs
[What was given up for what was gained]

## 14. Alternatives
[Other viable designs and why this one was chosen]

Every system design chapter in this Part (11.2–11.9) is, structurally, a filled-out version of this exact template.

18. Short exercise

A customer says: "We want an AI chatbot for our employees." Using only steps 1-4 of this chapter's process, write out at least five specific clarifying questions you'd need answered before beginning any architecture work — deliberately including at least two non-functional requirement questions the customer is unlikely to volunteer unprompted.

19. Interview questions

• Walk through your system design process for a new AI feature, and explain why you address failure modes, security, and cost as explicit steps rather than an afterthought.
• Why is the distinction between functional and non-functional requirements important, and give an example of a non-functional requirement that's commonly under-specified by customers.
• Why should alternatives and trade-offs be discussed only after the chosen design is fully worked out, not before?

20. FDE/customer scenario

Customer: "Just tell us what technology to use — we don't need the whole process, we already know we want [a specific trendy technology]."

The FDE-correct response gently but firmly insists on at least a lightweight pass through requirements and constraints first — not out of process for its own sake, but because Part 3.12's entire argument (over-engineering, under-engineering, and everything in between) shows that skipping straight to a technology choice, however trendy or well-intentioned, routinely produces expensive, hard-to-reverse mistakes that a brief requirements conversation would have caught early and cheaply.

---

Key takeaways

• System design should proceed through requirements and constraints first, architecture in the middle, and failure modes/security/cost/trade-offs as integral, non-negotiable final steps — never skipped or treated as an afterthought.
• Non-functional requirements are frequently under-specified by customers and must be actively surfaced, not assumed obvious.
• Every technology/architecture decision should trace back to a specific requirement or constraint — reasoning backward from a trendy technology choice is the anti-pattern this entire framework exists to prevent.

Things you should be able to explain

• The full 14-step system design process and why the ordering matters.
• The distinction between functional and non-functional requirements, with an example of each that's commonly missed.

Things you should be able to build

• A requirements-and-constraints intake template and a full system-design-record document following this chapter's structure.

Common mistakes

• Starting from a technology choice instead of requirements.
• Treating failure modes, security, and cost as an afterthought rather than integral design steps.
• Not surfacing unstated non-functional requirements.

Recommended next chapter

02-design-enterprise-rag.md