2.6 — LLMs (Large Language Models)

1. What is it?

A Large Language Model is a decoder-only transformer (Part 2.4) trained on massive text corpora to predict the next token in a sequence, scaled up in parameters and data to the point where it exhibits broad, general capabilities (following instructions, reasoning through problems, writing code, holding conversations) that weren't explicitly programmed. This chapter is the conceptual bridge from "AI/ML fundamentals" into Part 3's deep, practical AI engineering material.

2. Why does it exist?

Task-specific NLP models (Part 2.3) each required their own labeled dataset and training run — useful, but narrow and expensive to build one-per-task. LLMs exist because scaling up a single, generically-trained model (train on next-token prediction over an enormous, diverse corpus) produced a model capable of handling an enormous range of tasks it was never explicitly trained for, through prompting alone — a fundamentally different, far more flexible way of directing a model's behavior than the old "collect labels, train a classifier" loop.

3. What problem does it solve?

For an AI FDE, LLMs solve the "we need software that behaves flexibly across many different, hard-to-fully-specify tasks, without building and maintaining a separate model for each one" problem. This is precisely why LLMs are so central to enterprise AI: enterprise problems are messy, varied, and change often, and a general-purpose model that can be redirected via prompting (versus a narrow model that must be retrained) fits that reality much better.

4. How does it work internally?

Pretraining, then alignment

Modern LLMs are trained in stages, and understanding this pipeline explains a lot of observed behavior:

1. Pretraining:  massive, diverse text corpus → next-token prediction objective
                 (produces a "base model" — good at completing text, not at
                  following instructions or being helpful/harmless)
        │
        ▼
2. Instruction tuning (SFT): fine-tune on curated instruction/response examples
                 (teaches the model to behave like an assistant, not just
                  autocomplete arbitrary text)
        │
        ▼
3. Alignment (RLHF/RLAIF-style techniques): further tuning using human or
                 AI-generated preference signals, optimizing for helpfulness,
                 harmlessness, and following the intended behavior

A raw pretrained base model, if you could interact with one directly, doesn't reliably follow instructions — it just continues text in a statistically plausible way. The instruction-tuned, aligned "chat" model you actually call via an API is the product of this full pipeline, and its behavior (helpfulness, refusals, tone) is a direct consequence of the choices made at stages 2 and 3, not an inherent property of "being an LLM."

Autoregressive generation and sampling

As covered in Part 2.4, generation is token-by-token: the model outputs a probability distribution over the next token, and a sampling strategy decides which token is actually chosen.

• Temperature: scales the probability distribution before sampling — low temperature (near 0) makes the model nearly deterministic, favoring the highest-probability token; higher temperature flattens the distribution, increasing variety and randomness (and the chance of less coherent output).
• Top-p (nucleus sampling): samples only from the smallest set of tokens whose cumulative probability exceeds p, cutting off the long tail of unlikely tokens.
• Top-k: samples only from the k most likely tokens.

These parameters directly explain reproducibility (or lack of it): temperature 0 is the closest you get to deterministic output from most providers, but even at temperature 0, exact determinism isn't always guaranteed across all providers/hardware — a real, practical caveat when a customer expects "the exact same input always produces the exact same output," which is a stronger guarantee than most LLM APIs actually provide.

What "hallucination" actually is

An LLM has no separate, explicit fact database it consults — it generates the statistically plausible continuation of the prompt, based on patterns learned during training. When the plausible continuation happens to be factually wrong (a confidently stated fake citation, a wrong date), this is not a bug in the traditional sense — it's the expected behavior of a system that was never trained to have or check "truth," only to produce fluent, plausible, contextually appropriate text. This reframing matters enormously for setting a customer's expectations correctly: you don't "fix" hallucination the way you fix a software bug; you manage it via grounding (RAG, Part 3.5), constrained output, verification steps, and evaluation (Part 8).

Inference-time knobs you don't train but do control

Everything above (pretraining, instruction tuning, alignment) shapes a model's weights, fixed once training finishes. But how that fixed model is actually run — served to real traffic — involves a separate set of engineering decisions that don't change what the model knows or how it behaves, only how efficiently and cheaply it produces answers. Three of these come up constantly once you move from "calling an API" to reasoning about self-hosting or provider infrastructure, and it's worth knowing what they are conceptually before Part 18.16 gets into the operational math of sizing them:

Quantization is running a model's weights (and sometimes activations) at lower numerical precision than they were trained in — e.g., representing each parameter with 8 or 4 bits instead of the 16 or 32 bits used during training. A transformer's parameters (Part 2.4's attention and feed-forward weights) don't need full training-time precision to still produce useful outputs at inference time; most of that precision was needed for stable gradient updates during training, not for the forward pass alone. The payoff is smaller memory footprint and often faster inference; the cost is some degradation in output quality, whose size varies by model and quantization method and must be evaluated empirically for your use case, not assumed.

Batching is processing multiple requests' forward passes together rather than one at a time, because a GPU's parallel compute is wasted running a single small sequence through — grouping requests lets the hardware do far more useful work per unit of time. Since attention computation scales with sequence length (Part 2.4), and different requests in a batch have different lengths and arrive at different times, how you batch (waiting for a fixed group vs. dynamically admitting new requests mid-batch) has a real effect on both throughput and per-request latency — the specific mechanics are Part 18.16's job.

Mixture-of-Experts (MoE) is an architectural choice about the feed-forward layers inside the transformer (Part 2.4's block diagram): instead of one dense feed-forward network that every token passes through, an MoE model has many smaller "expert" feed-forward networks, and a learned router sends each token to only a handful of them. This decouples a model's total parameter count from the compute cost of processing any single token — a model can have a very large total parameter count (more learned capacity) while each forward pass only activates a fraction of it (closer to a smaller, cheaper model's compute cost per token). It's a different lever from quantization or batching: those change how a fixed architecture is run; MoE changes the architecture itself to get more capacity per unit of inference compute.

None of these change what this chapter has taught about pretraining, hallucination, or sampling — they're about the economics and throughput of serving a model that's already been trained and aligned. Operational depth (VRAM sizing, concrete throughput tradeoffs, when self-hosting a quantized or MoE model actually beats an external API) is in Part 18.16 — this chapter's job was only the conceptual "why these three levers exist."

Extended internal reasoning

Part 3's prompt-engineering material references "some current models perform extended internal reasoning by default" as an aside — worth introducing properly here, since it belongs with this chapter's other model-behavior material, not as an unexplained forward reference. Chain-of-thought prompting (asking a model to "think step by step" in its visible output) works because generating intermediate reasoning tokens gives the model more computation — each generated token can attend to every earlier one — before it commits to a final answer. Some current model families build a version of this in architecturally: the model performs an extended internal reasoning process before producing its user-facing response, without an engineer having to prompt for it explicitly, and (depending on the provider and product surface) that internal reasoning may be shown, summarized, or hidden entirely from the caller. This is not a different kind of model in the sense of a new architecture family — it's the same next-token-prediction transformer from this chapter, trained and/or prompted such that more of its own generated tokens are spent on intermediate reasoning before the final answer, rather than only on the answer itself. The practical consequence: for a model with this behavior, additional explicit chain-of-thought prompting may add much less marginal benefit than it would for a model without it — check current provider documentation for a given model rather than assuming CoT prompting has uniform value everywhere, and expect this reasoning to consume additional tokens (and therefore cost and latency, section 12), whether or not it's shown to the end user.

5. Simple mental model

An LLM is a extremely well-read improv actor, not a database. It has absorbed an enormous amount of patterns from what it's read, and it improvises the most plausible next line given the scene so far (the prompt) — brilliantly, most of the time, drawing on genuinely learned structure and patterns. But like any improviser, when it doesn't actually know the specific fact needed, it doesn't say "I don't know" by default — it improvises something plausible-sounding, because plausible continuation, not verified truth, is what it was trained to produce. Grounding it in real documents (RAG) is like handing the actor a script for the parts that must be factually exact.

6. Real-world example

A financial services customer asks their newly deployed assistant "what's the current interest rate on our savings product," and the model confidently states a plausible-sounding but outdated (or entirely fabricated) rate. This isn't a "broken model" — it's the direct, predictable consequence of asking a next-token predictor with no access to live data to answer a question requiring current, precise facts. The fix isn't "get a smarter model" (a fundamental limitation, not a capability gap) — it's retrieval-augmentation: give the model the actual current rate as context and instruct it to answer only from that (Part 3.5).

7. Architecture diagram

Pretrainingnext-token prediction on massive corpus

Instruction tuning (SFT)teach assistant behavior

AlignmentRLHF/RLAIF-style · helpful, harmless

Deployed "chat" model

A raw pretrained base model reaches your API only after two more training stages.

8. Production considerations

• Don't assume temperature=0 gives byte-for-byte reproducibility across all providers — verify this claim against the specific provider's current documentation if exact reproducibility is a hard customer requirement.
• Model behavior can shift across provider-side updates even at a nominally stable model identifier in some cases — track and pin model versions explicitly where the provider supports it, and monitor for behavior drift (Part 8.4).
• Understand and set correct expectations for "knowledge cutoff" — a model has no knowledge of events after its training data was collected, and won't reliably know this about itself unless told; this is directly relevant to any customer question involving recent events or current data (exactly what RAG and tool-calling, Part 3, exist to solve).

9. Common mistakes

• Treating hallucination as a solvable "bug" that a bigger/better model will eventually eliminate entirely, rather than an inherent property to be managed via architecture (grounding, verification, evaluation).
• Assuming the model "knows" things about the world in the way a database does, rather than generating plausible continuations based on learned patterns.
• Not distinguishing between the base model's raw capability and the specific behavior shaped by instruction tuning/alignment — e.g., assuming a model will refuse something because "LLMs are safe," when refusal behavior is a trained, adjustable property, not an immutable law.

10. Security considerations

• Because generation is fundamentally "produce a plausible continuation of the input," any content that becomes part of that input (including retrieved documents, tool outputs, or user messages) can influence the model's behavior — this is the direct mechanistic root of prompt injection (Part 9.1); it follows necessarily from how generation works, not from a specific implementation flaw.
• Alignment/safety training reduces but does not eliminate the ability to elicit unintended behavior through careful prompting (jailbreaking) — a genuinely adversarial, ongoing research area (Part 9), not a solved problem.

11. Performance considerations

• Latency has two distinct components worth measuring separately: time to first token (largely a function of prompt processing/prefill) and tokens per second during generation (largely a function of model size and generation length) — conflating these leads to misdiagnosing latency issues (Part 7 covers this in production detail).
• Output length directly drives both latency and cost — a model asked for a concise answer is faster and cheaper than one asked for an exhaustive one, independent of "how hard" the underlying question is.

12. Cost considerations

• Cost scales with both input tokens (the whole prompt/context, including retrieved documents and conversation history) and output tokens (often priced differently, output typically costing more per token than input) — Part 7.10 builds a full cost-optimization framework on this foundation.
• Bigger/more capable models cost more per token — model selection (Part 3.11) is fundamentally a capability/cost/latency trade-off, not "always use the best model."

13. When to use it

Tasks requiring flexible language understanding/generation, broad world knowledge (with the caveat of knowledge cutoff and hallucination risk), instruction-following across varied and evolving requirements — the enormous majority of "AI engineering" work today starts here.

14. When NOT to use it

• Tasks requiring guaranteed, current, or precise factual data — use grounding (RAG, tool calling) rather than relying on the model's parametric "knowledge" alone.
• Tasks requiring strict determinism/auditability where "usually correct" isn't good enough — pair with deterministic validation layers (Part 1.9, Part 9).
• Narrow, high-volume, latency-critical tasks where a classical model (Part 2.1, 2.3) is cheaper and sufficient.

15. Alternatives and trade-offs

Approach	Good for	Weak point
LLM (prompted)	Flexible, broad, fast to build	Hallucination risk, cost/latency scale with tokens, imperfect determinism
LLM + RAG (Part 3.5)	Grounded, current, verifiable answers	Added retrieval pipeline complexity
Classical ML/NLP	Narrow, cheap, deterministic, auditable	Doesn't generalize; needs retraining as needs evolve
Fine-tuned LLM (Part 2.2)	Consistent behavior/format for a narrow task	Training/maintenance overhead; not the right tool for injecting current facts

16. Practical Python/code example

from anthropic import AsyncAnthropic

client = AsyncAnthropic()


async def generate_grounded_answer(question: str, context: str) -> str:
    """
    Generates an answer constrained to the provided context, explicitly instructing
    the model to acknowledge when the context doesn't contain the answer, rather
    than filling the gap with a plausible-sounding guess.

    Args:
        question (str): The user's question.
        context (str): Retrieved, trusted context relevant to the question.

    Returns:
        str: The model's answer.
    """
    response = await client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=500,
        temperature=0,
        system=(
            "Answer only using the provided context. "
            "If the context does not contain the answer, say so explicitly instead of guessing."
        ),
        messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}],
    )
    return response.content[0].text

Note temperature=0 for consistency and an explicit system instruction addressing the hallucination-management strategy from section 4 directly — telling the model what to do when it doesn't have the answer, rather than leaving that behavior to chance.

17. Production-quality example

A wrapper that logs model/version/parameters alongside every generation — essential for debugging behavior drift (section 8) and for reproducing a customer-reported issue exactly:

import logging
import time
from dataclasses import dataclass

logger = logging.getLogger("llm_calls")


@dataclass
class GenerationRecord:
    """A fully-specified record of one LLM call, sufficient to reproduce or audit it later."""
    model: str
    temperature: float
    prompt_tokens: int
    completion_tokens: int
    latency_ms: float
    request_id: str


async def generate_with_audit_trail(
    client, model: str, system: str, user_message: str, request_id: str, temperature: float = 0.0
) -> tuple[str, GenerationRecord]:
    """
    Calls the LLM and returns both the response and a full audit record of the call.

    Args:
        client: An initialized async LLM client.
        model (str): The exact model identifier used, pinned rather than left implicit.
        system (str): System instructions.
        user_message (str): The user-facing message content.
        request_id (str): Correlation ID for tracing this call back to the originating request.
        temperature (float): Sampling temperature.

    Returns:
        tuple[str, GenerationRecord]: The generated text and its full audit record.
    """
    start = time.monotonic()
    response = await client.messages.create(
        model=model,
        max_tokens=1000,
        temperature=temperature,
        system=system,
        messages=[{"role": "user", "content": user_message}],
    )
    latency_ms = (time.monotonic() - start) * 1000

    record = GenerationRecord(
        model=model,
        temperature=temperature,
        prompt_tokens=response.usage.input_tokens,
        completion_tokens=response.usage.output_tokens,
        latency_ms=latency_ms,
        request_id=request_id,
    )
    logger.info("generation_record=%s", record)
    return response.content[0].text, record

18. Short exercise

A customer insists "the AI lied to us" when it confidently stated an incorrect policy detail. Write, in plain non-technical language suitable for that customer, a two-to-three sentence explanation of why this happened that doesn't use the word "hallucination" as an unexplained jargon term, and that points toward the actual fix (grounding).

19. Interview questions

• Explain the pretraining → instruction tuning → alignment pipeline and why a raw base model doesn't behave like a helpful assistant.
• Why is hallucination better understood as an inherent property to manage than a bug to eliminate?
• What's the practical difference between "time to first token" and "tokens per second," and why does distinguishing them matter for diagnosing latency?
• Conceptually, what does quantization trade away, and why does an MoE model decouple total parameter count from per-token inference cost?
• Why might a model with built-in extended reasoning need less explicit chain-of-thought prompting than one without it?

20. FDE/customer scenario

Customer: "Why can't you just make the model never make things up?"

The honest, technically-grounded answer explains that the model generates plausible continuations rather than looking up verified facts, so "never hallucinate" isn't a setting to enable — but grounding the model's answers in retrieved, trusted documents (RAG) and constraining it to only answer from that context (with explicit instructions to say "I don't know" otherwise) dramatically reduces the practical impact, and evaluation (Part 8) lets you measure and report exactly how often it still happens. This is a foundational conversation every AI FDE has, in some form, on nearly every engagement.

---

Key takeaways

• LLMs are the product of a pretraining → instruction-tuning → alignment pipeline; behavior comes from all three stages, not an inherent property of "being an LLM."
• Hallucination is a structural property of next-token generation, managed through grounding and evaluation, not eliminated by a bigger model alone.
• Sampling parameters (temperature, top-p/top-k) directly control the determinism/creativity trade-off.
• Quantization, batching, and MoE are inference-time/architectural levers on cost and throughput, separate from what training shapes; Part 18.16 covers their operational sizing.
• Some models perform extended internal reasoning by default, changing how much explicit chain-of-thought prompting adds.

Things you should be able to explain

• Why a raw pretrained model doesn't behave like a helpful assistant.
• Why hallucination happens mechanistically, in plain language a non-technical customer can follow.
• Conceptually why quantization, batching, and MoE each affect inference cost/throughput without changing what the model has learned.

Things you should be able to build

• A grounded-generation call with explicit "don't guess" instructions and a full audit trail of model/parameters/usage.

Common mistakes

• Treating hallucination as a bug a better model will simply fix.
• Assuming temperature=0 guarantees full determinism across all providers.
• Not tracking model version/parameters for reproducibility.
• Assuming CoT prompting adds equal value to every model regardless of built-in reasoning behavior.

Recommended next chapter

Part 2 complete. Continue to handbook/03-ai-engineering/01-prompt-engineering.md.