3.1 — Prompt Engineering

1. What is it?

Prompt engineering is the discipline of designing the input to an LLM — instructions, context, examples, format constraints — to reliably produce the output you want. It's "programming" a system whose behavior is controlled through natural language rather than code, which makes it a genuinely different kind of engineering discipline than what most software engineers are trained for.

2. Why does it exist?

An LLM's behavior is enormously sensitive to how a task is framed: the same underlying request, phrased differently, structured differently, or given different examples, can produce meaningfully different quality output. Prompt engineering exists because this sensitivity is real and consequential — treating prompt-writing as an afterthought ("just ask it nicely") produces unreliable, hard-to-debug systems, while treating it as an engineering discipline (versioned, tested, iterated) produces reliable ones.

3. What problem does it solve?

It solves "how do I get consistent, reliable behavior out of a fundamentally non-deterministic, general-purpose system." Since you already have practical prompting experience (per this book's assumptions), this chapter focuses less on basic technique and more on the engineering discipline around prompting: how prompts fail in production, how to structure them for maintainability, and how to reason about them the way you'd reason about any other critical piece of application logic.

4. How does it work internally?

Why prompt structure affects output at all

Recall from Part 2.4 that a model generates the statistically plausible continuation of its input. Prompt structure works because it changes what "plausible continuation" means: a prompt phrased as a formal technical question tends to pull the model toward patterns learned from technical writing in its training data; a prompt with few-shot examples in a specific format strongly biases the model toward continuing in that same format, because that's now the most locally plausible continuation. Prompting isn't magic — it's steering the conditional probability distribution the model samples from, by changing what's actually in the context.

The core techniques and why each works

• Zero-shot: instructions alone, no examples. Works when the task is common enough in training data that the model reliably generalizes from the instruction.
• Few-shot: providing 2-5 examples of input→output pairs. Works by making the desired pattern the overwhelmingly locally-plausible continuation — especially valuable for output format consistency, less so for teaching genuinely new capabilities the model doesn't have.
• Chain-of-thought (CoT): instructing the model to reason step-by-step before answering. Works because generating intermediate reasoning tokens gives the model "more computation" to work with (each generated token can be attended to by subsequent tokens) before committing to a final answer — genuinely improves performance on multi-step reasoning tasks, not just a formatting preference. Note: some current models perform extended internal reasoning by default/architecture (varies by model and provider — verify current behavior), which changes how much explicit CoT prompting adds; check current model documentation rather than assuming CoT prompting is equally impactful across all models.
• Role/persona framing ("You are an expert..."): works by conditioning the plausible-continuation distribution toward patterns associated with that framing in training data — real effect, but easy to overstate; a persona doesn't grant new capability, it shifts style/tone/framing.

System vs. user vs. assistant messages

Most modern chat APIs structure input as role-tagged messages (system, user, assistant), not raw concatenated text. The system message typically carries the highest-priority, most persistent instructions (behavior, constraints, persona); user/assistant messages carry the actual conversation turns. Providers give system messages elevated weight/priority in how the model is trained to respect them, which is why critical constraints belong there rather than buried in a user message — but this is a trained behavioral tendency, not an absolute technical guarantee, which is directly relevant to why prompt injection (Part 9.1) remains possible even with well-structured system prompts.

5. Simple mental model

Prompting is giving directions to a brilliant, extremely literal-minded new employee who has read almost everything ever written but has no memory of your specific business and no way to ask clarifying questions mid-task. Vague directions get plausible-sounding but often wrong results; specific, structured directions with examples of what "good" looks like get reliable results — this isn't different from managing any new hire's first week, except you can't have a follow-up conversation mid-task.

6. Real-world example

A support-ticket triage prompt initially just said "categorize this ticket." In production, categories were inconsistent ("billing" vs "Billing" vs "billing issue"), silently breaking a downstream routing system expecting exact string matches. Adding few-shot examples with the exact target format, plus an explicit enumerated list of valid categories in the system prompt, eliminated the inconsistency — a direct illustration of few-shot prompting solving a format-reliability problem, not a capability problem (the model always "understood" the categories; it just wasn't reliably formatting its answer).

7. Architecture diagram

System messagerole, hard constraints, output format

Few-shot examplesshape output format/style

Retrieved contextif RAG · grounds factual answers

Conversation historyuser/assistant turns

Current user message

Assembled together, top to bottom, this is "the prompt" sent to the model.

8. Production considerations

• Version and test prompts like code (Part 1.7, Part 1.9) — a prompt change is a behavior change to a production system, and should go through the same review/eval discipline as a code change.
• Separate the stable "template" from the variable content — a prompt template with clearly parameterized slots (user input, retrieved context) is far easier to reason about, test, and diff than a prompt reconstructed ad hoc via string concatenation scattered through the codebase.
• Build a regression eval set (Part 8) before making prompt changes to a live system — "it seems better on the three examples I tried" is not sufficient evidence for a production change.
• Explicitly specify output format (JSON schema, enumerated categories, length constraints) rather than hoping the model infers your intended format — ambiguity here is a leading cause of downstream parsing failures (Part 3.2 covers structured output enforcement in depth).

9. Common mistakes

• Iterating on prompts by "trying it a few times and eyeballing the result" without a regression eval set, then being surprised when a change that looked like an improvement regresses a different case in production.
• Burying critical constraints deep in a long user message instead of the system message, where they're more likely to be deprioritized against later context.
• Assuming a technique that worked well on one model (e.g., a specific CoT phrasing) transfers identically to a different model or model version without re-testing — prompting techniques are not fully model-agnostic.
• Over-engineering prompts with excessive meta-instructions ("you must always," "never under any circumstances") that don't actually improve reliability and just add token cost — measure, don't guess, whether an added instruction actually changes behavior.

10. Security considerations

• Anything placed in the prompt — including retrieved documents and tool outputs — is part of what the model conditions on, meaning untrusted content in the prompt is a potential injection vector (Part 9.1). Prompt engineering and prompt-injection defense are two sides of the same underlying mechanism.
• Don't rely on a system-prompt instruction alone ("never reveal your system prompt," "never do X") as your only defense against a determined adversarial user — these instructions raise the bar but are not an absolute guarantee (Part 9 covers layered defenses).

11. Performance considerations

• Longer prompts (more few-shot examples, more instructions) cost more latency and money per call (Part 2.4, Part 2.6) — there's a real trade-off between "more examples for reliability" and "faster/cheaper calls," worth measuring rather than assuming more is always better.
• Chain-of-thought prompting that generates substantial reasoning text increases output tokens (and thus cost/latency) — worth evaluating whether the reliability gain justifies the cost for a given task, especially for models with built-in extended reasoning where explicit CoT prompting may add less marginal benefit.

12. Cost considerations

• Every fixed element of a prompt template (system instructions, few-shot examples) is paid for on every single call — a bloated, rarely-revisited system prompt is a hidden, recurring cost multiplied across your entire request volume (Part 7.10 quantifies this).

13. When to use it

For directing any LLM's behavior — this is the default, foundational skill underlying every other AI engineering technique in this book.

14. When NOT to use it (i.e., when prompting alone isn't the fix)

• When the task requires facts the model doesn't have or that must be current — that's a grounding (RAG) problem, not a prompting problem (Part 3.5).
• When the task requires a genuinely new capability the model can't do reliably at all, regardless of framing — that may indicate a need for a different/better model (Part 3.11), fine-tuning (Part 2.2), or restructuring the task (breaking it into steps, adding tools) rather than more prompt tweaking.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Prompt engineering alone	Fast iteration, no training needed	Can't add facts the model doesn't have or fix a genuine capability gap
Prompting + RAG	Grounds answers in current/specific facts	Added retrieval pipeline complexity
Fine-tuning	Consistent behavior/format at scale, potentially lower per-call cost	Slower iteration, training/maintenance overhead, wrong tool for facts

16. Practical Python/code example

from dataclasses import dataclass


@dataclass
class TicketTriagePrompt:
    """A versioned, parameterized prompt template for support ticket triage."""

    VALID_CATEGORIES = ["billing", "technical", "account", "other"]

    SYSTEM = (
        "You triage support tickets into exactly one of these categories: "
        f"{', '.join(VALID_CATEGORIES)}. "
        "Respond with only the category name, lowercase, nothing else."
    )

    FEW_SHOT_EXAMPLES = [
        {"role": "user", "content": "I was charged twice this month"},
        {"role": "assistant", "content": "billing"},
        {"role": "user", "content": "The app crashes when I upload a file"},
        {"role": "assistant", "content": "technical"},
    ]

    def build_messages(self, ticket_text: str) -> list[dict]:
        """
        Builds the full message list for triaging a ticket.

        Args:
            ticket_text (str): The raw support ticket text.

        Returns:
            list[dict]: Messages ready to send to the LLM API, in role-tagged format.
        """
        return [*self.FEW_SHOT_EXAMPLES, {"role": "user", "content": ticket_text}]

17. Production-quality example

A prompt version registry with an enforced eval gate, tying together Part 1.7 (Git) and Part 1.9 (Testing) with prompt engineering discipline:

import logging
from dataclasses import dataclass, field

logger = logging.getLogger("prompt_registry")


@dataclass
class PromptVersion:
    """A single, immutable, versioned prompt definition."""
    version: str
    system: str
    few_shot: list[dict] = field(default_factory=list)
    min_eval_score: float = 0.9


class PromptRegistry:
    """Tracks prompt versions and enforces that only eval-passing versions are marked active."""

    def __init__(self):
        self._versions: dict[str, PromptVersion] = {}
        self._active_version: str | None = None

    def register(self, prompt: PromptVersion, eval_score: float) -> None:
        """
        Registers a new prompt version, promoting it to active only if it clears
        the required evaluation threshold.

        Args:
            prompt (PromptVersion): The prompt version to register.
            eval_score (float): The score this version achieved on the regression eval set.

        Raises:
            ValueError: If the eval score is below the version's minimum threshold.
        """
        if eval_score < prompt.min_eval_score:
            raise ValueError(
                f"prompt version {prompt.version} scored {eval_score:.3f}, "
                f"below required {prompt.min_eval_score}; not activating"
            )
        self._versions[prompt.version] = prompt
        self._active_version = prompt.version
        logger.info("activated prompt version=%s eval_score=%.3f", prompt.version, eval_score)

    def get_active(self) -> PromptVersion:
        """Returns the currently active, eval-passing prompt version."""
        if self._active_version is None:
            raise RuntimeError("no prompt version has been activated yet")
        return self._versions[self._active_version]

18. Short exercise

Take the TicketTriagePrompt example and add a fifth category, "security", along with two new few-shot examples. Then write, in plain language, what you'd check in a regression eval set before deploying this change, specifically to catch a scenario where the new category accidentally "steals" tickets that should still be classified as "technical".

19. Interview questions

• Why does chain-of-thought prompting actually improve performance on multi-step reasoning tasks, mechanistically?
• Why should prompts be versioned and eval-gated the same way code is?
• Explain why a persona/role instruction ("you are an expert lawyer") changes style/framing but doesn't grant new factual knowledge.

20. FDE/customer scenario

Customer: "Can you just tweak the prompt to make it stop doing X?"

This is one of the most common requests an AI FDE gets, and the correct response resists the instinct to just edit the prompt live and declare victory. The right process: reproduce the failure case, check it against the existing regression eval set (or add it if missing), make the change, and re-run the full eval set before deploying — because a prompt change that fixes the reported case can easily regress a different, previously-working case, and "seems fixed" without evaluation is not a safe basis for a production change.

---

Key takeaways

• Prompt structure works by shaping what continuation is statistically plausible, not through magic — understanding this demystifies technique selection.
• Prompts are production code and deserve the same versioning, testing, and review discipline.
• Prompting alone can't add missing facts (use RAG) or grant a genuinely missing capability (consider a different model or restructuring the task).

Things you should be able to explain

• Why few-shot examples improve format reliability specifically.
• Why chain-of-thought prompting genuinely improves reasoning performance, not just output verbosity.

Things you should be able to build

• A versioned prompt registry with an enforced evaluation gate before activation.

Common mistakes

• Iterating on prompts without a regression eval set.
• Burying critical constraints in user messages instead of the system message.
• Assuming prompting techniques transfer identically across models without re-testing.

Recommended next chapter

02-structured-outputs.md