11.3 — System Design: Customer Support Agent

Scenario: A mid-market e-commerce company wants a customer-facing AI support agent that can answer order-status questions, process straightforward returns, and escalate complex issues to human agents — handling roughly 50,000 conversations/month with significant daily/seasonal traffic variance (e.g., major spikes around holiday shopping).

1. Requirements

Reduce human support team's workload for routine questions (order status, simple returns) while maintaining customer satisfaction and never taking an irreversible action (a large refund, an account change) without appropriate safeguards.

2. Constraints

• Existing order/customer data lives in the company's e-commerce platform and a separate CRM (Part 10.3).
• Support volume is highly seasonal — must handle 5-10x normal traffic during peak shopping periods without degrading response quality or availability.
• Customer trust is paramount — a visibly broken or unhelpful bot experience directly damages the brand relationship, arguably worse than not having a bot at all.

3. Functional Requirements

• Order status lookup and tracking information (Part 3.3's tool-calling, real-time data — explicitly NOT a RAG problem, Part 3.5, section 14).
• Simple return processing within defined bounds (Part 9.2's calibrated autonomy).
• Escalation to a human agent with full conversation context handed off cleanly.
• FAQ-style policy questions (return policy, shipping times) — a good RAG fit (Part 3.5), distinct from the real-time lookup functionality above.

4. Non-Functional Requirements

• p95 latency under 3 seconds for a typical response (customer-facing, higher bar than the internal-tool example in 11.2).
• Must handle 5-10x traffic spikes without degradation (Part 7.5's scaling discipline, with particular attention to Part 7.5's bottleneck-diagnosis: is scaling bound by infrastructure or LLM provider rate limits during a spike).
• No unauthorized access to another customer's order/account data (Part 9.3/9.6's tenant/user-level isolation, here at the individual-customer level rather than organizational-tenant level).

5. Architecture

Customer message

Routing workflowintent classification, fast/cheap model (Part 3.7/3.11)

Order statustool call, real-time (Part 3.3)

FAQ/PolicyRAG (Part 3.5)

Return processingbounded workflow, human approval above threshold (Part 5.4/9.2)

Human escalationwith full context handoff — any path, if confidence low or customer requests it

6. Components

• Intent-classification router (Part 3.7's routing pattern): a small, fast model (Part 3.11) directs each message to the appropriate fixed handler, keeping the overall system a workflow rather than an open-ended agent (Part 3.7's core preference) for the majority of routine cases.
• Order-status tool: direct, real-time tool call to the e-commerce platform's API (Part 3.3/10.3), never RAG, since order status is precise, structured, frequently-changing data.
• FAQ RAG pipeline: for genuinely static policy content (Part 3.5), kept separate from the real-time order-status path.
• Bounded return-processing workflow: small returns process autonomously (within a defined dollar/reason-code threshold, Part 9.2's hard-coded limits), larger or unusual returns route to human-in-the-loop approval (Part 5.4).
• Escalation handoff: packages full conversation history and context for the receiving human agent, avoiding the customer having to repeat themselves.

7. Data Flow

1. Customer message arrives via chat widget.
2. Intent classifier routes to order-status, FAQ, return-processing, or (if confidence is low) directly to escalation.
3. Order-status path: tool call to e-commerce platform API, real-time result, natural-language response.
4. FAQ path: RAG retrieval over policy documents, grounded generation.
5. Return path: check against autonomous-processing thresholds (Part 9.2); if within bounds, execute directly; if not, pause for human approval (Part 5.4).
6. Any path can escalate to a human agent at any point, with full conversation history attached.

8. Failure Modes

• E-commerce platform API unavailable: order-status tool returns a graceful "unable to look up your order right now, let me connect you with a team member" rather than a technical error or a hallucinated guess (Part 2.6/3.3).
• Traffic spike exceeding provisioned capacity: autoscaling (Part 7.4/7.5) with a defined fallback to a lighter-weight, cheaper model (Part 3.11) if the primary model's rate limit is being approached, preserving availability over full capability during extreme peaks.
• Return-processing tool bug: bounded by the hard-coded threshold limits (Part 9.2) regardless of any agent reasoning error — the worst-case damage from any single mistake is capped by design.

9. Security

Excessive-agency mitigation (Part 9.2) is central given the return-processing capability's real financial consequence — hard-coded autonomous-processing limits, human-in-the-loop above threshold. Per-customer data isolation (Part 9.3/9.6) prevents one customer's conversation from ever surfacing another's order/account data, enforced via user-scoped tool construction (Part 3.3/4.4).

10. Scalability

Seasonal 5-10x traffic spikes are the central scaling challenge (constraint 2) — addressed via Part 7.4/7.5's autoscaling and Part 3.11's model-routing fallback to a smaller/cheaper model under extreme load, plus Part 7.9's gateway-level rate-limit coordination if this system shares LLM provider quota with other internal systems.

11. Observability

Task-completion rate and escalation rate as key quality SLIs (Part 8.1/8.4) — a rising escalation rate is an early signal of either a real product issue or an emerging edge case the routing/handlers don't cover well. Cost-per-conversation tracked explicitly (Part 7.10), with particular attention during traffic spikes when cost can scale disproportionately if model-routing fallbacks aren't functioning correctly.

12. Cost

Primary drivers: LLM calls for the intent classifier (high volume, should use the smallest viable model, Part 3.11) and generation calls for FAQ/order-status responses. Seasonal spike cost planning (Part 7.10) should explicitly model the 5-10x traffic multiplier, including the potential for a fallback to a cheaper model during peak load actually reducing average per-conversation cost during exactly the highest-volume period.

Worked numeric example: at 50,000 conversations/month, assume each conversation involves one cheap intent-classification call (~150 tokens, a small model at roughly $0.25/million input, $1.25/million output ≈ negligible, ~$0.0002/call) plus one generation call (~500 input + 150 output tokens on a mid-tier model at $3/$15 per million ≈ $0.0015 + $0.00225 ≈ $0.00375/call). Normal-month cost ≈ 50,000 × ($0.0002 + $0.00375) ≈ $189/month. During a 10x holiday spike concentrated into roughly a third of the month's days, daily volume can reach ~16,700 conversations/day; average conversations/month during peak season stay bounded by the same per-conversation cost as long as the cheaper-model fallback (section 12) engages under load — modeling the spike explicitly, rather than only the monthly average, is what catches whether the fallback is actually needed to keep peak-day cost proportionate rather than spiking 10x alongside volume.

At a 10x spike, throughput needed is roughly 50,000 × 10 / 30 days / 86,400 seconds ≈ 0.19 conversations/second sustained, with real intra-day concentration during peak shopping hours pushing well above this average — the actual design-relevant number for capacity planning (Part 7.5) is the peak-hour rate, not the monthly average.

13. Trade-offs

Chose a workflow-based architecture (fixed routing) over a fully open-ended agent (Part 3.7) for the majority of traffic, trading some flexibility for materially better predictability, auditability, and cost control — appropriate given most support conversations genuinely follow a small number of known patterns. Chose hard-coded thresholds for autonomous return processing over trusting agent judgment for all cases, trading some autonomy for a hard, provable safety bound (Part 9.2).

14. Alternatives

A fully autonomous, open-ended agent handling all support scenarios (no fixed routing) was considered and rejected — Part 3.7's core argument applies directly: most support conversations have genuinely fixed, knowable structure, and a workflow captures this more reliably and auditably than an agent re-deriving the right approach every time. Revisit if a significant fraction of conversations genuinely require open-ended, unpredictable multi-step handling the current routing categories don't capture well.

15. Code Example

The bounded, hard-coded return-processing threshold check (section 6/9's central excessive-agency mitigation) — the model's judgment informs the request, but never single-handedly authorizes it above the threshold:

from dataclasses import dataclass


@dataclass
class ReturnRequest:
    order_id: str
    refund_amount_usd: float
    reason_code: str


AUTONOMOUS_REFUND_LIMIT_USD = 100.00
AUTONOMOUS_REASON_CODES = {"wrong_item", "damaged_in_transit", "duplicate_order"}


def resolve_return(request: ReturnRequest) -> str:
    """
    Determines whether a return can be processed autonomously or must be
    escalated for human approval, per a hard-coded threshold that no
    model output can override.

    Args:
        request (ReturnRequest): The customer's return request.

    Returns:
        str: "auto_approved" if within the autonomous-processing bounds,
            otherwise "pending_human_approval".
    """
    within_amount_limit = request.refund_amount_usd <= AUTONOMOUS_REFUND_LIMIT_USD
    within_allowed_reason = request.reason_code in AUTONOMOUS_REASON_CODES
    if within_amount_limit and within_allowed_reason:
        return "auto_approved"
    return "pending_human_approval"

16. Interview questions

• Given 50,000 conversations/month and a 10x holiday spike, walk through estimating both average and peak-hour cost/throughput, and explain why the peak-hour number, not the monthly average, drives capacity planning.
• Why does the return-processing threshold need to be a hard-coded check rather than a rule the model itself is simply instructed to follow?

17. FDE/customer scenario

CUSTOMER: "The agent is smart enough — just let it decide refund amounts on its own, we trust the model."

The FDE-correct response holds the line on the hard-coded threshold (section 6/9) regardless of demonstrated model quality — Part 9.2's excessive-agency principle is that a capability's worst-case bound should be provable from the code, not from confidence in the model's typical behavior, since "typical" and "worst case" are different questions and only the threshold answers the second one.

---

Key takeaways

• Order-status lookups are a tool-calling problem, not a RAG problem — precise, structured, real-time data belongs behind a direct tool call (Part 3.3), never embedded and retrieved semantically.
• Return processing's real financial consequence makes excessive-agency mitigation (hard-coded thresholds, human-in-the-loop above them) the central security design decision.
• Seasonal traffic spikes require both infrastructure autoscaling and a cost-aware model-routing fallback, planned explicitly rather than discovered under real peak load.

Things you should be able to explain

• Why order-status lookup should never be built as a RAG feature.
• Why hard-coded thresholds, not agent judgment alone, bound the return-processing capability's risk.

Things you should be able to build

• A routing workflow combining real-time tool calls, RAG, and a bounded, human-in-the-loop-gated action capability.

Common mistakes

• Building real-time structured lookups as RAG instead of tool calls.
• Trusting agent judgment alone for financially consequential actions instead of hard-coded limits.

Recommended next chapter

04-design-multi-tenant-ai-saas.md