13.2 — Rapid Prototyping and Demos

1. What is it?

Rapid prototyping is building the smallest, fastest, cheapest version of a proposed solution specifically to validate the riskiest assumptions before committing to full production investment (Part 13.1's step 3). A demo is the specific artifact used to communicate a prototype's results — and, done well, to build stakeholder confidence and buy-in for the fuller build that follows.

2. Why does it exist?

Part 12.2 established that feasibility should be validated, not assumed. Rapid prototyping is the concrete practice that does this validation — cheaply and quickly, before Part 13.1's much larger step-4 production investment. It exists because the cost of discovering a fundamental flaw (a feasibility gap, a misunderstood requirement, a technical approach that doesn't actually work as expected) grows enormously the later it's discovered — cheap and fast in a two-day prototype, expensive and slow in a three-month production build already underway.

3. What problem does it solve?

It solves "how do I validate the riskiest, least-certain parts of a proposed solution as cheaply and quickly as possible" — deliberately not trying to build a complete, production-quality system, but targeting exactly the specific uncertainties (Part 12.2's feasibility questions) that would be most costly to discover late.

4. How does it work internally?

Identifying the riskiest assumption — the prototype's actual target

A prototype should not attempt to validate everything at once — it should target the single (or few) assumption(s) whose failure would most seriously undermine the whole project, identified explicitly before any prototype code is written:

For the underwriting-automation example (Part 11.2/12.2):
  Riskiest assumption: "Can the model reliably extract the required
  fields from real underwriting documents at the accuracy level our
  business case depends on?"
  → NOT the riskiest assumption: "Can we build a nice UI for this"
    (low uncertainty — obviously yes, just execution time)
  → NOT the riskiest assumption: "Can we deploy this on Kubernetes"
    (low uncertainty — well-understood, standard practice)

Prototype scope: a script that runs structured extraction (Part 3.2)
against 30-50 REAL sample documents (never synthetic ones — Part 12.2's
feasibility-probe example used exactly this pattern), measuring actual
accuracy against known-correct answers — nothing else. No UI, no
authentication, no production infrastructure, no polish.

This is a direct, deliberate application of Part 12.2's feasibility-probe pattern (section 17's exact code example) — a prototype's entire value comes from answering the specific, highest-uncertainty question as fast and cheaply as possible, not from looking like a finished product.

Prototype quality bar — deliberately, explicitly lower than production

A prototype is allowed — should be expected — to skip everything this handbook's Parts 1, 7, and 9 require for production: no comprehensive error handling, no authentication, no horizontal scaling, no full security hardening. This is not laziness; it's a deliberate, explicit trade-off, and communicating this distinction clearly to stakeholders (Part 12.3's communication discipline) is essential — a stakeholder who mistakes a prototype for a near-finished product will have miscalibrated expectations about both timeline and remaining risk.

Demos — communicating validated (or invalidated) assumptions honestly

A demo should show real results against real (or realistic) inputs, honestly — including failure cases, not just the cherry-picked successes that make for an impressive but misleading show. An FDE's credibility depends on this honesty: a demo that hides known limitations produces a customer who's surprised and disappointed later, exactly the trust-damaging outcome Part 12's entire discovery discipline exists to prevent.

5. Simple mental model

Rapid prototyping is like a chef tasting a new recipe's key, uncertain ingredient combination in a small test batch before committing to cooking it for 200 wedding guests — the test batch doesn't need to be plated beautifully or served at the right temperature; its entire purpose is answering "does this specific, uncertain part actually work" cheaply, before the much larger, much more expensive full commitment. A demo is like honestly sharing that test batch's actual results with the wedding client — including if the combination didn't quite work as hoped, so decisions can be made with real information, rather than confidently serving a beautifully-presented dish made from an untested recipe to 200 guests and hoping for the best.

6. Real-world example

Before committing to the three-month production build in Part 13.1's real-world example, the FDE team's two-week prototype targeted exactly one question: could structured extraction reliably pull the required fields from a sample of 40 real underwriting documents at the accuracy the business case required. The prototype — a single Python script with no UI, no database, no deployment infrastructure — ran extraction against these real documents and measured accuracy directly (Part 12.2's exact feasibility-probe pattern), revealing 91% field-level accuracy on the first attempt, comfortably above the required 85% threshold, and specifically identifying which document sub-type (documents with an unusual, less-common formatting variant, about 8% of the sample) accounted for most of the remaining errors — concrete, specific information that directly shaped the full production build's design (a fallback path for that specific document sub-type) rather than a vague "extraction generally seems to work" impression from informal testing.

7. Architecture diagram

Identify the SINGLE riskiest assumptionNOT everything — the specific highest-uncertainty question

Build the SMALLEST possible testtargeting ONLY that question — skip UI, auth, scaling, full error handling deliberately

Measure against REAL sample datanot synthetic, honestly

Demo the ACTUAL resultssuccesses AND failures — informs the go/no-go decision (Part 13.1)

8. Production considerations

• Explicitly identify the riskiest assumption before writing any prototype code — section 4's discipline; a prototype without this focus risks becoming an unfocused, half-built mini-version of the whole system rather than a sharp answer to the question that actually matters.
• Communicate the prototype/production quality gap explicitly to stakeholders — a stakeholder should never mistake prototype code for near-production-ready work (section 4's communication point).
• Use real, representative sample data, never synthetic or cherry-picked data — Part 12.2's feasibility-probe pattern depends entirely on this; a prototype validated against easy, unrepresentative examples gives false confidence.
• Show failure cases honestly in demos, not just successes — section 4's trust-preserving practice, directly connected to Part 12's discovery-trust discipline.

9. Common mistakes

• Attempting to validate everything in one prototype instead of targeting the single riskiest, highest-uncertainty assumption specifically.
• Building prototype code to a near-production quality bar, wasting time on polish (Part 1/7's production concerns) that a prototype's actual purpose doesn't need yet.
• Testing against synthetic or easy, cherry-picked sample data rather than real, representative cases — producing a falsely reassuring result that doesn't hold up once real production data is encountered.
• Demoing only successful cases, hiding known limitations, and creating a customer whose expectations don't match reality once the full system is built.

10. Security considerations

A prototype explicitly skipping production-grade security (Part 9) must never handle real, sensitive production data carelessly just because "it's only a prototype" — even a throwaway prototype touching real customer data still needs baseline data-handling discipline (Part 9.6's minimization principle), scoped appropriately to the prototype's actual, temporary purpose.

11. Performance considerations

A prototype's performance characteristics (Part 7) are typically irrelevant to what it's actually validating (usually a capability/accuracy question, not a scale question) — don't spend prototype time optimizing performance unless performance itself is the specific riskiest assumption being tested (a genuinely different, rarer case).

12. Cost considerations

Rapid prototyping is fundamentally a risk-reduction, cost-avoidance practice — a two-week prototype costing a small fraction of a three-month production build's cost, specifically to avoid discovering a fatal flaw only after the full investment (section 6's real numbers illustrate this concretely).

13. When to use it

Before any significant production investment where a genuine, high-consequence uncertainty exists (typically feasibility, Part 12.2) — essentially every real FDE engagement with a novel or unproven-for-this-customer technical approach.

14. When NOT to use it

A well-understood, previously-validated pattern being applied again for a very similar customer/use case may not need fresh prototyping — the uncertainty a prototype exists to resolve may already be resolved by prior, directly-transferable experience (Part 13.1's lifecycle-loop point).

15. Alternatives and trade-offs

The "alternative" to rapid prototyping is committing directly to full production build based on assumed feasibility — section 6 shows the concrete value prototyping provided (both confirming feasibility AND revealing a specific, addressable weakness) that a skipped-prototype approach would have discovered only much later, at much higher cost.

16. Practical Python/code example

A focused, single-question prototype script — deliberately minimal, targeting exactly one risky assumption, extending Part 12.2's feasibility-probe pattern:

import logging

logger = logging.getLogger("prototype")


async def run_extraction_prototype(client, real_sample_documents: list[dict]) -> dict:
    """
    A deliberately minimal prototype validating ONE specific question:
    can structured extraction reliably pull required fields from real
    documents at an acceptable accuracy — nothing else is built or tested.

    Args:
        client: An async LLM client.
        real_sample_documents (list[dict]): REAL documents with known-correct
            expected extractions, gathered during discovery (Part 12.1/12.2).

    Returns:
        dict: Overall accuracy plus a breakdown by document sub-type, to
            identify SPECIFIC weaknesses, not just an aggregate pass/fail.
    """
    results_by_subtype: dict[str, list[bool]] = {}

    for doc in real_sample_documents:
        extracted = await extract_fields(client, doc["text"])
        correct = extracted == doc["expected_fields"]
        subtype = doc.get("subtype", "standard")
        results_by_subtype.setdefault(subtype, []).append(correct)

    breakdown = {
        subtype: sum(results) / len(results)
        for subtype, results in results_by_subtype.items()
    }
    overall_accuracy = sum(sum(r) for r in results_by_subtype.values()) / len(real_sample_documents)

    logger.info("prototype accuracy: overall=%.2f breakdown=%s", overall_accuracy, breakdown)
    return {"overall_accuracy": overall_accuracy, "breakdown_by_subtype": breakdown}

Note the breakdown by document sub-type — directly what surfaced the specific, actionable weakness in section 6's real-world example, rather than a single aggregate number that would have hidden it.

17. Production-quality example

A prototype-results demo summary, structured to present results honestly per section 8's recommendation:

# Prototype Results: Underwriting Document Extraction

## What we tested
Structured field extraction against 40 real underwriting documents from
your archive, measuring accuracy against your team's own verified answers.

## Results
- Overall accuracy: 91% (target: 85%+) — FEASIBLE
- Standard-format documents (92% of sample): 96% accuracy
- Non-standard-format documents (8% of sample): 61% accuracy — a
  SPECIFIC, IDENTIFIED gap requiring a fallback path in the full build

## What this means for the full build
We recommend proceeding with production development, with an explicit
fallback path (routing non-standard-format documents to human review,
Part 5.4) rather than attempting full automation for that specific
8% sub-type until we've gathered enough examples to improve accuracy
there specifically.

## What this prototype does NOT tell us
This prototype did not test: system performance at scale, integration
with your existing systems, or the full user experience — those are
addressed in the full production build.

The explicit "what this prototype does NOT tell us" section is a deliberate honesty practice — proactively bounding what was and wasn't validated, rather than letting a successful demo imply more confidence than the prototype's actual scope supports.

18. Short exercise

A customer asks for a prototype of an AI system that (a) extracts data from documents and (b) has a polished, branded user interface, within a one-week timeline. Using this chapter's principles, explain which of these two things you'd prioritize building and why, and how you'd communicate that prioritization decision to the customer.

19. Interview questions

• Explain how you'd identify the "riskiest assumption" to target in a prototype, using a concrete example.
• Why should a prototype deliberately skip production-quality engineering (error handling, scaling, security hardening), and how do you communicate this gap to stakeholders?
• Why is testing against real sample data, rather than synthetic or cherry-picked data, essential to a prototype's actual value?

20. FDE/customer scenario

CUSTOMER: "This prototype demo looked great — can we just deploy this to production next week?"

The FDE-correct response is honest and direct: the prototype validated the specific, riskiest technical question (feasibility) deliberately by skipping everything else a production system needs (Part 1, 7, 9's full engineering discipline) — explaining this distinction clearly, with the specific list of what's still needed (security hardening, error handling, scaling, integration, Part 13.1's step 4), prevents exactly the kind of miscalibrated-expectation problem this chapter's section 4/8 warned against, and is a genuine trust-building moment when handled with this level of honesty rather than an over-eager "sure, let's ship it."

---

Key takeaways

• A prototype should target the single riskiest, highest-uncertainty assumption specifically, not attempt to validate everything at once.
• Prototype code deliberately, explicitly skips production-quality engineering — this gap must be communicated clearly to stakeholders, never left implicit.
• Demos should show real results honestly, including failure cases and explicit scope limitations, preserving the trust that Part 12's entire discovery discipline is built on.

Things you should be able to explain

• How to identify the riskiest assumption a prototype should target.
• Why prototype quality is deliberately lower than production quality, and why this must be communicated explicitly.

Things you should be able to build

• A focused, single-question prototype script with sub-type-level accuracy breakdown, and an honest, scope-bounded demo results summary.

Common mistakes

• Trying to validate everything in one prototype instead of the single riskiest assumption.
• Building prototype code to near-production quality, wasting time on unnecessary polish.
• Demoing only successes, hiding known limitations from stakeholders.

Recommended next chapter

Part 13 complete. Continue to handbook/14-deployment/01-deployment-in-customer-environments.md.