2.4 — Transformers

1. What is it?

The transformer is the neural network architecture (Vaswani et al., "Attention Is All You Need," 2017) underlying every modern LLM. Its core mechanism, self-attention, lets a model weigh the relevance of every other token in a sequence when processing each token — in parallel, rather than one token at a time.

2. Why does it exist?

Before transformers, sequence models (RNNs, LSTMs) processed text token-by-token, sequentially, maintaining a compressed hidden state. This made them slow to train (can't parallelize across the sequence — each step depends on the previous one) and bad at capturing long-range dependencies (information from far earlier in a long sequence tends to fade from the hidden state). Transformers replaced sequential recurrence with attention, which lets every token look directly at every other token regardless of distance, and — critically — lets all tokens be processed in parallel during training, which is what made training on today's massive datasets and model sizes computationally feasible.

3. What problem does it solve?

It solves "how does a model figure out which parts of a long input actually matter for understanding/generating the current part" — without this, "The trophy didn't fit in the suitcase because it was too big" is genuinely hard for a model to resolve (what does "it" refer to?) without being able to directly relate "it" back to "trophy" across the intervening words, regardless of distance.

4. How does it work internally?

Self-attention, conceptually

For each token, the model computes three vectors: a Query (what am I looking for), a Key (what do I offer), and a Value (what information do I actually carry). Attention for a given token is computed by comparing its Query against every other token's Key (via dot product), turning those comparisons into weights (via softmax), and producing a weighted sum of all tokens' Values:

Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · V

Concretely: to resolve "it" in the trophy/suitcase sentence, "it"'s Query vector ends up having high similarity with "trophy"'s Key vector (learned from training data patterns), so the attention weight on "trophy" is high, and "trophy"'s Value contributes strongly to "it"'s updated representation.

Multi-head attention

Rather than computing attention once, transformers compute it multiple times in parallel ("heads"), each potentially learning to attend to different kinds of relationships (one head might specialize in syntactic dependencies, another in coreference, another in something with no clean human label) — the results are concatenated and combined. This gives the model multiple, simultaneous "lenses" on the same sequence.

The full block

Input embeddings+ positional encoding

Multi-head self-attention

Feed-forward network

Output representationsblock repeats N times

One transformer block; GPT-scale models stack dozens of these.

Positional encoding exists because attention itself has no inherent notion of order (it's a set operation over tokens) — position information is injected explicitly so the model knows "the" came before "cat," not just that both appeared.

Residual connections (adding a layer's input to its output) and layer normalization are what make training networks this deep stable — without them, gradients tend to vanish or explode across dozens of stacked layers (Part 2.2).

Encoder-only, decoder-only, encoder-decoder

• Encoder-only (e.g., BERT-family): sees the whole input at once, bidirectionally — good for understanding/classification tasks, not natural for open-ended generation.
• Decoder-only (e.g., the GPT/Claude/most modern LLM family): generates one token at a time, each token only attending to previous tokens (causal/masked attention) — this is the architecture behind essentially every chat-style LLM you'll use as an AI FDE.
• Encoder-decoder (e.g., original translation transformers, T5): an encoder processes the full input, a decoder generates output attending back to the encoder's representations — common for translation/summarization-specific models, less common as the base for general-purpose chat LLMs today.

Autoregressive generation

A decoder-only LLM generates text one token at a time: predict a probability distribution over the next token, sample/select one, append it to the sequence, repeat — using the growing sequence (including its own previous outputs) as context each time. This is why generation is inherently sequential at inference time (unlike training, which parallelizes across the whole sequence) and why longer outputs take proportionally longer to generate.

5. Simple mental model

Self-attention is like a room full of people simultaneously deciding how much to listen to each other to understand a shared conversation: each person (token) has something they're trying to figure out (Query), something they can offer (Key/Value), and everyone updates their understanding by weighting everyone else's contribution based on relevance — all at once, not one person talking at a time.

6. Real-world example

When an LLM answers a question using a long document you've pasted into the prompt, attention is the mechanism that lets it "find" the relevant sentence anywhere in that document and connect it to your question — regardless of whether that sentence was near the beginning or deep in the middle. This is also exactly why very long contexts can still have effective retrieval degradation ("lost in the middle" — models empirically attend less reliably to information buried in the middle of very long contexts, an active area of ongoing model improvement, not a settled solved problem) — a real, practical reason RAG (Part 3.5) often outperforms just "stuffing everything into a huge context window," even when the context window is technically large enough.

7. Architecture diagram

"The trophy didn't fit in the suitcase because it was too big"

Attention weights for "it" (illustrative):
  The       trophy    didn't    fit     in    the   suitcase  because   it   was  too   big
  0.02       0.71      0.01    0.03    0.01   0.01    0.12      0.01     -   0.01 0.02  0.05
                ▲                                        ▲
        high attention weight                    some attention weight
        ("it" resolves to "trophy")               (context word)

8. Production considerations

• Attention's compute cost scales roughly quadratically with sequence length in the standard formulation — this is a direct, structural reason why very long contexts are more expensive and slower, and why context-window management (what you actually put in the prompt) is a real engineering discipline, not an afterthought (Part 3.1, Part 7.10).
• "Lost in the middle" effects mean prompt/context ordering matters — placing the most critical information near the start or end of a long context is a practical mitigation many teams use, though the effect and best mitigation strategy varies by model and continues to improve across model generations — don't treat this as a fixed, permanent rule; verify against current model behavior.
• Causal (decoder-only) architectures mean a model's output at position N never sees tokens generated after position N — relevant for understanding why streaming generation can't "go back and fix" an earlier mistake mid-generation without external orchestration (e.g., re-prompting).

9. Common mistakes

• Assuming a large context window means information anywhere in it is retrieved equally reliably — empirically not always true, and a common source of "why did it miss something that was clearly in the prompt" debugging sessions.
• Not accounting for attention's cost scaling with context length when reasoning about latency/cost at scale.
• Confusing "the model has a huge context window" with "RAG is unnecessary" — context window size and effective retrieval quality/cost are separate concerns (Part 3.5 addresses this trade-off directly).

10. Security considerations

• Because attention lets any part of the input influence any part of the output, adversarial or injected content anywhere in a long context (a malicious instruction buried in a retrieved document, for instance) can influence the model's behavior — this is the architectural root of indirect prompt injection (Part 9.1), not an incidental implementation bug.

11. Performance considerations

• Techniques like KV-caching (reusing previously computed Key/Value vectors for earlier tokens instead of recomputing them at every generation step) are what make autoregressive generation practically fast — a detail that matters when reasoning about why the first token of a response often takes longer than subsequent ones ("time to first token" vs. "tokens per second," both real metrics in LLM gateway/latency work, Part 7).

12. Cost considerations

• Quadratic-ish attention cost with sequence length is a direct driver of why longer prompts/contexts cost more — not just from the token-pricing multiplication, but from the underlying compute cost providers pass through in pricing.

13. When to use it

You don't choose "to use a transformer" directly as an AI FDE — you choose among transformer-based models (Part 3.11 covers model selection). Understanding the architecture is what lets you reason correctly about context-length trade-offs, latency, and failure modes rather than treating the model as a total black box.

14. When NOT to use it

Not applicable in the same sense — nearly every current LLM you'll integrate is transformer-based. The judgment call is architecture-adjacent: encoder-only models for pure classification/embedding tasks (Part 2.5) versus decoder-only generative models for chat/agent tasks — picking the wrong family for the task is a real, avoidable mistake.

15. Alternatives and trade-offs

Architecture family	Good for	Weak point
Decoder-only transformer (GPT/Claude-style)	General-purpose generation, chat, agents	Autoregressive generation is inherently sequential at inference
Encoder-only transformer (BERT-style)	Classification, embeddings, understanding tasks	Not naturally suited to open-ended generation
Encoder-decoder transformer	Translation, summarization-specific tasks	Less commonly the base for general-purpose chat LLMs today
Pre-transformer RNN/LSTM	Historically useful, still relevant in some low-resource/streaming contexts	Sequential training, weaker long-range dependency handling

16. Practical Python/code example

A minimal, illustrative (not production-scale) self-attention computation, to make the math concrete:

import numpy as np

def self_attention(query: np.ndarray, key: np.ndarray, value: np.ndarray) -> np.ndarray:
    """
    Computes scaled dot-product self-attention for a small set of token vectors.

    Args:
        query (np.ndarray): Query vectors, shape (seq_len, d_k).
        key (np.ndarray): Key vectors, shape (seq_len, d_k).
        value (np.ndarray): Value vectors, shape (seq_len, d_v).

    Returns:
        np.ndarray: Attention-weighted output, shape (seq_len, d_v).
    """
    d_k = query.shape[-1]
    scores = query @ key.T / np.sqrt(d_k)
    weights = np.exp(scores) / np.exp(scores).sum(axis=-1, keepdims=True)  # softmax
    return weights @ value

17. Production-quality example

Given that you rarely implement attention yourself, the production-relevant skill is reasoning about context placement to mitigate lost-in-the-middle effects — a prompt-assembly function that puts the highest-priority content at the start/end of the context window:

def assemble_context(
    system_instructions: str,
    critical_facts: list[str],
    supporting_documents: list[str],
    user_question: str,
    max_supporting_docs: int = 10,
) -> str:
    """
    Assembles an LLM prompt with the most critical information placed at the
    start and end of the context, mitigating empirically observed degraded
    attention to content buried in the middle of long contexts.

    Args:
        system_instructions (str): Core behavioral instructions for the model.
        critical_facts (list[str]): Facts that must not be missed (e.g., account status,
            active policy exceptions) — placed immediately before the question.
        supporting_documents (list[str]): Lower-priority retrieved context.
        user_question (str): The user's actual question.
        max_supporting_docs (int): Cap on how many supporting documents to include.

    Returns:
        str: The assembled prompt text.
    """
    docs = "\n\n".join(supporting_documents[:max_supporting_docs])
    facts = "\n".join(f"- {fact}" for fact in critical_facts)
    return (
        f"{system_instructions}\n\n"
        f"SUPPORTING CONTEXT:\n{docs}\n\n"
        f"CRITICAL FACTS (verify your answer against these):\n{facts}\n\n"
        f"QUESTION: {user_question}"
    )

Placing critical_facts right before the question (near the end of the context, close to where generation begins) is a deliberate mitigation, not arbitrary ordering.

18. Short exercise

Explain, using the Query/Key/Value framing, why the word "it" in "The trophy didn't fit in the suitcase because it was too big" ends up attending strongly to "trophy" rather than "suitcase," in terms of what the model has learned during training (you don't need the literal numbers — describe the mechanism).

19. Interview questions

• Explain self-attention using the Query/Key/Value framework, and why it replaced sequential recurrence.
• What is "lost in the middle," and what are two practical mitigations for it in prompt design?
• Why does decoder-only causal attention make autoregressive generation inherently sequential, and how does KV-caching address the resulting performance cost?

20. FDE/customer scenario

Customer: "We just pasted our entire 300-page policy manual into the prompt since the model supports a huge context window — why does it still miss things that are clearly in there?"

This is a direct, practical consequence of attention behavior at long context lengths (section 6/8). The FDE-correct response isn't "the model is broken" — it's explaining that context window size and reliable retrieval are different properties, and proposing either RAG (retrieve only the relevant sections per query, Part 3.5) or restructuring the prompt to place the most relevant sections strategically, rather than assuming a bigger context window alone solves the retrieval problem.

---

Key takeaways

• Self-attention lets every token weigh every other token's relevance directly, in parallel — the core innovation over sequential RNNs.
• Decoder-only, causal transformers are the architecture behind essentially all modern chat/agent LLMs.
• Long contexts have real, empirically observed retrieval-reliability limits ("lost in the middle"), independent of nominal context-window size.

Things you should be able to explain

• The Query/Key/Value mechanism at a conceptual level.
• Why attention cost scales with sequence length and what that implies for cost/latency.

Things you should be able to build

• A context-assembly function that strategically places critical information to mitigate lost-in-the-middle effects.

Common mistakes

• Assuming a large context window guarantees reliable retrieval of anything placed in it.
• Ignoring context-length cost/latency scaling when designing prompts.

Recommended next chapter

05-embeddings.md