2.3 — NLP (Natural Language Processing)

1. What is it?

NLP is the field concerned with getting computers to process, understand, and generate human language. Before LLMs, this meant a patchwork of specialized techniques (tokenizers, part-of-speech taggers, named-entity recognizers, sentiment classifiers, translation models) — each often a separately trained model for a narrow task. Understanding this pre-LLM landscape matters because much of it still exists in production systems and still shows up as components inside modern AI pipelines.

2. Why does it exist?

Language is unstructured, ambiguous, and context-dependent in ways that plain string processing can't handle: "bank" means different things in "river bank" and "bank account," word order matters, and meaning depends on things far outside the current sentence. NLP as a field exists to build the specific techniques for handling this — from simple statistical methods to today's transformer-based LLMs.

3. What problem does it solve?

For an AI FDE, classical NLP concepts solve two ongoing problems: (1) many production pipelines still use lightweight, fast, non-LLM NLP components for tasks that don't need a full LLM (language detection, basic tokenization, simple entity extraction with a regex/rule-based system) because they're cheaper and faster, and (2) understanding tokenization specifically is foundational to reasoning about LLM cost, context limits, and behavior (Part 2.6).

4. How does it work internally?

Tokenization

Before any model — classical or LLM — can process text, it must be converted into discrete units (tokens). Modern LLMs use subword tokenization (e.g., byte-pair encoding, BPE, or similar variants) — not whole words, not individual characters, but frequently-occurring subword chunks, learned from a large corpus.

"unhappiness" → ["un", "happi", "ness"]     (illustrative — actual tokenizer output varies)
"ChatGPT"     → ["Chat", "G", "PT"]          (uncommon words often split into more tokens)

This is why token count doesn't map cleanly to word count, why unusual words/names/non-English text often "cost" more tokens than expected, and why a customer's cost estimate based on "words" will be systematically wrong (Part 3, Part 7.10 build on this directly for cost engineering).

Classical pipeline stages (still relevant)

Raw text → Tokenization → POS tagging → Named Entity Recognition → downstream task
              (spaCy,          (identify         (identify people,
               NLTK, etc.)      grammatical        organizations,
                                 role of each       dates, amounts)
                                 token)

A classical Named Entity Recognition (NER) model, trained specifically to tag entities, is often faster, cheaper, and more deterministic than asking an LLM to extract entities via prompting — a real, current trade-off in production pipeline design (Part 3.12).

Embeddings as the bridge to modern NLP

Word2Vec and GloVe (pre-transformer) were the first widely-used techniques to represent words as dense vectors capturing semantic similarity ("king" - "man" + "woman" ≈ "queen"), directly foreshadowing the embedding-based retrieval (Part 2.5, Part 3.4) that underlies modern RAG systems. The core idea — represent meaning as position in a continuous vector space — has been central to NLP for over a decade before "embeddings" became an AI-engineering buzzword.

5. Simple mental model

Classical NLP is a toolbox of specialized single-purpose instruments (a tokenizer, a POS tagger, an NER model, a sentiment classifier), each good at exactly one narrow job, that you'd wire together into a pipeline. An LLM is closer to a generalist instrument that can approximate most of those jobs through prompting alone, at the cost of being slower, more expensive, and less predictable per call than a purpose-built classical tool.

6. Real-world example

A logistics company needs to redact PII (names, addresses, phone numbers) from customer messages before they're stored for training data. A classical NER-based redaction pipeline is fast, cheap, runs entirely offline/on-prem (a real requirement for privacy-sensitive customers), and — crucially — is deterministic and auditable: the same input always produces the same redaction decision, which matters for compliance sign-off in a way that "usually redacts correctly" from an LLM does not.

7. Architecture diagram

Raw text

Tokenizer

Task-specific modelNER / sentiment / classifier / LLM

Structured outputentities, labels, generated text

Classical NLP swaps in a purpose-built model at the same slot an LLM occupies here.

8. Production considerations

• Choose classical NLP components over an LLM call when the task is narrow, high-volume, latency-sensitive, and doesn't need broad language understanding — e.g., language detection, basic PII redaction, simple keyword/rule-based routing.
• Tokenizer mismatch is a real, subtle production bug: if you estimate context-window usage with the wrong tokenizer (e.g., estimating for one model family but calling another), your token-count math will be wrong, potentially causing truncation or unexpected costs.
• Classical models generally need periodic retraining as language/data drifts (Part 2.1); LLMs used via API don't have this specific maintenance burden but have their own version-drift concerns (a provider updating a model can shift behavior even at the "same" model name/version in some cases — verify against provider changelogs).

9. Common mistakes

• Assuming "word count" and "token count" are interchangeable when estimating LLM cost or context-window usage.
• Reaching for an LLM for narrow, deterministic tasks (language detection, exact keyword matching) where a classical, cheaper, faster, and more auditable tool would serve better.
• Not considering that names/technical terms/non-English text tokenize less efficiently, causing surprising cost or truncation issues for certain customer content (e.g., a customer with many non-English-language documents may see meaningfully higher token counts than English-only estimates suggested).

10. Security considerations

• Classical, deterministic redaction/anonymization pipelines are often required (not just preferred) for regulated data before it can touch an LLM at all — an LLM-based redaction step is not deterministic enough to be the sole safeguard for genuinely sensitive PII in many compliance contexts (Part 9.6).

11. Performance considerations

• Classical NLP models (small, purpose-built) run in milliseconds on CPU; LLM calls run in hundreds of milliseconds to seconds over a network — a meaningful latency difference when a pipeline has many sequential steps.

12. Cost considerations

• A classical model, once trained/deployed, has near-zero marginal per-call cost (just compute); an LLM API call has a real, per-token marginal cost — at high volume, this difference compounds significantly, and is a legitimate reason to keep or introduce classical NLP components in a high-throughput pipeline rather than replacing everything with LLM calls (Part 7.10).

13. When to use it

Narrow, high-volume, latency-sensitive, or compliance-sensitive text-processing tasks with well-defined scope: tokenization (always, implicitly, whenever you call an LLM), language detection, deterministic PII redaction, simple classification at scale.

14. When NOT to use it

Open-ended understanding, generation, reasoning, or tasks requiring broad world knowledge or flexible instruction-following are exactly where LLMs now dominate over building/training a bespoke classical model.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Classical NLP (spaCy, regex, trained classifiers)	Fast, cheap, deterministic, auditable, narrow tasks	Doesn't generalize; needs retraining as data drifts; can't do open-ended reasoning
LLM via prompting	Broad, flexible, no training data needed, handles open-ended tasks	Slower, costlier per call, non-deterministic
Hybrid pipeline	Best of both — classical for narrow/high-volume steps, LLM for the genuinely open-ended step	More components to build, test, and maintain

16. Practical Python/code example

import tiktoken

def estimate_token_count(text: str, model_name: str = "gpt-4") -> int:
    """
    Estimates the token count of a text string for a given model family.

    Args:
        text (str): The text to tokenize.
        model_name (str): The model whose tokenizer should be used for the estimate —
            using the wrong model's tokenizer can meaningfully misestimate cost.

    Returns:
        int: The estimated number of tokens.
    """
    encoding = tiktoken.encoding_for_model(model_name)
    return len(encoding.encode(text))

Note: verify the current tokenizer library/encoding name against the provider you're actually calling — tokenizer details are provider- and model-family-specific and do change across model generations.

17. Production-quality example

A hybrid pre-processing pipeline: classical, deterministic PII redaction before any LLM call, with logging for audit purposes:

import logging
import spacy

logger = logging.getLogger("pii_redaction")
_nlp = spacy.load("en_core_web_sm")

_REDACTABLE_LABELS = {"PERSON", "GPE", "ORG", "PHONE", "EMAIL"}


def redact_pii(text: str) -> tuple[str, int]:
    """
    Redacts detected PII entities from text using a deterministic NER model,
    before the text is allowed to reach any LLM call.

    Args:
        text (str): Raw input text that may contain PII.

    Returns:
        tuple[str, int]: The redacted text, and the count of entities redacted
            (logged for audit trail purposes).
    """
    doc = _nlp(text)
    redacted = text
    redacted_count = 0
    for ent in sorted(doc.ents, key=lambda e: e.start_char, reverse=True):
        if ent.label_ in _REDACTABLE_LABELS:
            redacted = redacted[: ent.start_char] + f"[REDACTED_{ent.label_}]" + redacted[ent.end_char :]
            redacted_count += 1

    if redacted_count:
        logger.info("redacted %d PII entities before LLM call", redacted_count)
    return redacted, redacted_count

Iterating entities in reverse start-position order avoids offset shifts corrupting later replacements — a real, easy-to-get-wrong detail in text-splicing code like this.

18. Short exercise

A customer's cost estimate assumed 1 token ≈ 1 word for a corpus that's 40% non-English technical documentation. Explain why their actual token count (and thus cost) will likely be higher than estimated, and roughly how you'd produce a more accurate estimate before committing to a cost projection.

19. Interview questions

• Why doesn't token count map cleanly to word count, and why does this matter for cost estimation?
• When would you choose a classical NER model over an LLM prompt for entity extraction?
• Why might deterministic, classical redaction be a compliance requirement even in an LLM-heavy pipeline?

20. FDE/customer scenario

Customer: "Can the AI just detect and remove all personal information before anything gets logged?"

The FDE-correct answer separates two different claims: "an LLM can approximately do this" versus "this needs to be deterministic and auditable for compliance." If the customer's actual requirement is regulatory (GDPR, HIPAA-adjacent), the honest recommendation is a classical, deterministic redaction step as the enforced boundary, with an LLM only ever seeing already-redacted text — not "trust the LLM to redact well most of the time."

---

Key takeaways

• Tokenization (subword-based) is why token count doesn't equal word count — foundational to cost/context reasoning.
• Classical NLP components remain the right choice for narrow, high-volume, deterministic, or compliance-sensitive tasks.
• Embeddings/vector-space representations of meaning predate and directly foreshadow modern RAG.

Things you should be able to explain

• Why token count and word count diverge, and why it matters for cost.
• When a classical NLP component beats an LLM call.

Things you should be able to build

• A deterministic PII redaction step that runs before any LLM call.

Common mistakes

• Estimating LLM cost using word count instead of actual tokenization.
• Relying on an LLM as the sole safeguard for compliance-critical redaction.

Recommended next chapter

04-transformers.md