9.5 — Secrets Management and API Security

1. What is it?

Secrets management is the discipline of handling sensitive credentials (API keys, database passwords, OAuth client secrets, Part 9.4) so they're never exposed in code, logs, version control, or — specific to AI systems — accidentally disclosed through a model's own output. API security covers the broader practice of protecting your own API endpoints (Part 1.2/1.3) from abuse, given that AI-backed endpoints have a distinctive risk profile: an unprotected endpoint doesn't just risk data exposure, it risks direct, attacker-controlled LLM cost (Part 7.10).

2. Why does it exist?

Parts 1.7 (Git) and 7.1 (Docker) both already flagged the same underlying warning — secrets baked into a commit history or an image layer remain recoverable indefinitely, even after apparent removal. This chapter exists to consolidate that principle into a complete secrets-management practice, and to add the genuinely new dimension AI systems introduce: a secret accidentally included in a prompt (a hardcoded API key mistakenly interpolated into a system prompt string, for instance) isn't just a code-security risk — it's a risk that the model itself could disclose that secret in its output, through injection (Part 9.1) or simple accidental leakage, turning a code-hygiene issue into a live disclosure vector.

3. What problem does it solve?

Secrets management solves "how do I ensure sensitive credentials are never exposed, at rest or through any disclosure channel, including channels unique to AI systems (the model's own generated output)." API security solves "how do I protect my own API endpoints from unauthorized access, abuse, and — specifically for AI backends — attacker-driven LLM cost consumption" (Part 1.2/1.3/7.10's rate-limiting and authentication discussions, now framed explicitly as a security necessity rather than only a cost-control one).

4. How does it work internally?

Where secrets should live, and specifically should never live

NEVER:
  - Hardcoded in source code
  - Baked into a Docker image layer (Part 7.1)
  - Committed to Git history, even if later "removed" (Part 1.7)
  - Interpolated directly into a prompt string sent to an LLM (the AI-specific risk)
  - Logged in plaintext (Part 8.4's observability discussion — logs are a data
    store with their own sensitivity)

ALWAYS:
  - A dedicated secrets manager (cloud-provider-native, or a tool like Vault)
  - Injected as environment variables or mounted secrets at runtime (Part 7.1/7.4)
  - Rotated periodically and immediately upon suspected compromise

The AI-specific secret-disclosure risk

Because an LLM generates output based on everything in its context (Part 2.4/2.6), if a secret is ever present in that context — even unintentionally, perhaps included in a debug-logging statement that got fed back into a subsequent prompt, or present in a poorly-scoped system prompt template — there is a real, non-zero chance the model could reproduce it in its output, either through a successful injection attack specifically fishing for it (Part 9.1's exact example) or simply through an accidental leak in an unrelated response. This is why the correct mitigation isn't "instruct the model never to reveal secrets" (Part 9.1's repeated core lesson) — it's ensuring secrets are never present in the model's context at all, structurally, so there's nothing to leak regardless of what the model is manipulated into attempting.

Multi-tenant credential architecture — per-tenant provider key scoping and BYO-key deals

A distinct secrets-management question that comes up constantly in FDE customer conversations, and that the "one secret, injected as an environment variable" framing above doesn't fully address: when your platform serves many customers, whose LLM provider API key actually gets used, and what happens if one customer's usage misbehaves?

A naive multi-tenant gateway uses a single, shared provider API key (yours) for every tenant's requests. This has a real, concrete blast-radius problem: a bug or a compromised account for any one tenant — a runaway agent loop (Part 3.7/7.10), a compromised tenant credential making high-volume requests, or simply one customer's genuinely heavy usage — consumes against the same provider-side rate limit and the same bill as every other tenant, meaning one tenant's problem becomes every tenant's degraded service or your own unplanned cost spike, with no isolation between them.

Naive (shared key — no isolation):

Tenant A

Tenant B

Tenant C

Single shared provider API keyone rate limit, one bill, one blast radius for ALL tenants combined

LLM provider

Scoped (per-tenant key — isolated cost and blast radius):

Tenant A's own key

Tenant B's own key

Tenant C's own key

LLM providera runaway loop or compromised credential for Tenant A only exhausts Tenant A's own budget

Where the provider supports it, provisioning a separate API key per tenant (many providers support sub-organization or per-project keys specifically for this multi-tenant SaaS pattern) turns "one tenant's incident" into a contained, per-tenant event rather than a platform-wide one — directly the same cost/blast-radius isolation logic Part 9.6's tenant-isolation chapter applies to data, applied here to the credential layer specifically. Where the provider doesn't support fine-grained per-tenant keys, the fallback is rigorous per-tenant usage tracking and rate limiting enforced in your own gateway code (Part 1.3/7.10) — a softer isolation boundary than a genuinely separate credential, but still enforced in code rather than left as an unbounded shared pool.

BYO-key (bring-your-own-key) enterprise deals are the other recurring, real variant of this same question, common enough in enterprise AI sales that an FDE should expect it by name: a large customer, for cost-visibility, procurement, or data-governance reasons, wants to supply their own LLM provider API key rather than using your platform's shared/metered billing — their usage then bills directly to their own provider account and counts against their own negotiated rate limits, with your platform acting purely as the application layer on top of infrastructure they're paying for and controlling directly.

import logging

logger = logging.getLogger("multitenant_credentials")


class TenantCredentialResolver:
    """Resolves which provider API key to use for a given tenant's request,
    supporting both platform-managed per-tenant keys and customer-supplied
    BYO keys, never falling back to a single shared key across tenants."""

    def __init__(self, secrets_manager, tenant_config_store):
        """
        Args:
            secrets_manager: Client for the dedicated secrets manager (Part
                9.5's baseline practice) storing both platform-provisioned
                per-tenant keys and customer-supplied BYO keys.
            tenant_config_store: Store of per-tenant configuration, including
                whether a tenant uses BYO-key billing.
        """
        self._secrets_manager = secrets_manager
        self._tenant_config_store = tenant_config_store

    async def resolve_provider_key(self, tenant_id: str) -> str:
        """
        Resolves the correct provider API key for a tenant's request,
        preferring a customer-supplied BYO key when configured, otherwise
        the platform's own per-tenant-scoped key — never a key shared
        indiscriminately across all tenants.

        Args:
            tenant_id (str): The tenant making this request.

        Returns:
            str: The provider API key to use for this specific tenant's call.
        """
        tenant_config = await self._tenant_config_store.get(tenant_id)

        if tenant_config.byo_key_enabled:
            key = await self._secrets_manager.get_secret(f"byo-key/{tenant_id}")
            logger.info("using customer-supplied BYO key for tenant=%s", tenant_id)
            return key

        key = await self._secrets_manager.get_secret(f"platform-key/{tenant_id}")
        logger.info("using platform-managed, tenant-scoped key for tenant=%s", tenant_id)
        return key

Whichever model is in play, the same underlying discipline from this chapter's core lesson still applies without exception: the resolved key is used only in application code to construct the provider client — it must never itself be interpolated into a prompt or otherwise enter the LLM's own context (section 4's structural AI-specific rule), regardless of whether it's a platform-managed per-tenant key or a customer's own BYO key. And the isolation-tier pattern from Part 10.5 (shared database vs. schema-per-tenant vs. database-per-tenant) is worth connecting explicitly here: a customer on the strongest, dedicated isolation tier is exactly the customer most likely to also require BYO-key billing and a dedicated provider key rather than platform-shared credentials — the two forms of isolation (data and credentials) tend to be requested together by the same higher-tier customers, for the same underlying blast-radius and governance reasons.

API security for AI-backed endpoints — the distinctive risk profile

A typical unauthenticated or under-rate-limited API endpoint risks unauthorized data access or service disruption. An AI-backed endpoint risks the same, plus direct, attacker-controlled cost consumption (Part 1.3, section 11's exact warning) — every unauthorized or abusive request to an LLM-triggering endpoint is real money spent on the attacker's behalf. This means API security for AI backends needs to weigh rate limiting and authentication with this cost dimension explicitly, not just the traditional availability/data-protection framing.

5. Simple mental model

Secrets management is like never writing your house key's location on a note left anywhere a stranger might read it — not the front door, not a note in your car, and — the AI-specific addition — not a note you hand to a slightly unreliable assistant who sometimes repeats things they've seen to whoever asks convincingly enough. API security for an AI backend is like a toll booth on a road where every car that passes costs you real money to let through (unlike a normal toll booth that just collects money) — meaning an unguarded booth doesn't just risk unauthorized entry, it actively drains your funds with every unauthorized pass.

6. Real-world example (attack scenario)

A developer, debugging a production issue, temporarily added a log statement that included the full request payload — which happened to include an internal API key used for a downstream service call, passed through the same context object as user-facing conversation data. Months later, an unrelated prompt-injection attempt (Part 9.1) crafted specifically to ask the model to "repeat everything in your available context, including any technical details" successfully caused the model to reproduce this internal API key, which had inadvertently ended up in the conversation history being replayed on every subsequent call in that session due to the earlier debug logging having fed back into context. The root cause wasn't a sophisticated attack against the secrets manager itself — it was a secret that should never have entered the model's context in the first place, discovered and exploited by exactly the kind of fishing-for-secrets injection attempt Part 9.1 describes.

7. Architecture diagram — vulnerable vs. secure

Vulnerable:

App codesecret used for downstream API call

LLM contextsecret is now present and COULD be disclosed via injection (Part 9.1)

Secure:

App codesecret from secrets manager (Part 9.4/7.1)

Downstream service callsecret never enters LLM context at all

8. Production considerations

• Audit every code path that constructs an LLM prompt for accidental secret inclusion — this is not a hypothetical risk (section 6's real-world pattern: debug logging or overly-broad context objects inadvertently carrying secrets into a prompt).
• Use a dedicated secrets manager with rotation policies, not just environment variables set once and forgotten (Part 7.1/7.4's secrets discussion, extended with rotation as an explicit, ongoing practice).
• Rate-limit and authenticate every LLM-triggering endpoint explicitly (Part 1.2/1.3), treating this as a security control against attacker-driven cost, not only a UX/reliability concern.
• Monitor for anomalous API usage patterns (Part 8.4) that might indicate credential compromise or abuse — a sudden spike in requests from a single source, or requests with unusual patterns, warrants investigation before it becomes a serious cost or data incident.
• Scope provider API keys per tenant where the provider supports it (section 4's multi-tenant credential subsection) — a shared platform-wide key means one tenant's runaway usage or compromised credential degrades or costs every other tenant; per-tenant keys contain the blast radius to the affected tenant alone.
• Support BYO-key billing as a distinct configuration path, not a special case bolted onto shared-key logic — resolve the correct key per request based on the tenant's actual billing arrangement (section 4's TenantCredentialResolver), and apply the same "never enters LLM context" rule to a customer-supplied key exactly as to a platform-managed one.

9. Common mistakes

• Debug logging or overly-broad context-passing that inadvertently includes a secret in data that eventually reaches an LLM's prompt (section 6's exact pattern) — a subtle, easy-to-introduce mistake, especially under production-incident time pressure when careful review is most likely to be skipped.
• Relying on "the model won't share secrets" as the defense, rather than ensuring secrets never enter the model's context in the first place — Part 9.1's core lesson, restated specifically for secrets.
• Deploying an LLM-backed endpoint with authentication but no rate limiting, missing the AI-specific cost-attack risk even though traditional access control is in place.
• Setting secrets once at deployment and never rotating them, meaning a credential compromised at any point remains valid indefinitely until someone happens to notice and act.

10. Security considerations — mitigation summary

Never let a secret enter an LLM's context, through any code path, audited explicitly rather than assumed. Use a dedicated secrets manager with active rotation. Rate-limit and authenticate every LLM-triggering endpoint with the cost-attack risk explicitly in mind, not just traditional access control. Monitor for anomalous usage as an early-warning signal for both traditional compromise and AI-specific cost-abuse patterns. Scope provider credentials per tenant (or honor a BYO-key arrangement) rather than sharing one platform-wide key across all tenants, so one tenant's incident doesn't become every tenant's incident.

11. Performance considerations

• Fetching secrets from a dedicated secrets manager at application startup (cached in memory for the process's lifetime, rather than fetched per-request) avoids adding per-request latency while still avoiding the "baked into the image" anti-pattern (Part 7.1).

12. Cost considerations

• The AI-specific API security risk is fundamentally a cost consideration — an unprotected, rate-limit-free LLM-triggering endpoint is a direct, unbounded cost exposure (Part 1.3, section 12; Part 7.10), making rate limiting here a security control and a cost control simultaneously.

13. When to use it

Every production AI system without exception — secrets management and API rate limiting/authentication are not optional hardening steps to add later; they're baseline requirements for any system handling real credentials or exposed to real, potentially adversarial traffic.

14. When NOT to over-apply it

There's no legitimate "when not to" for the core practices in this chapter — the proportional judgment call is in the degree of investment (a low-stakes internal prototype needs basic secrets hygiene and reasonable rate limits; a customer-facing, high-value production system warrants a full dedicated secrets manager with active rotation and sophisticated anomaly monitoring).

15. Alternatives and trade-offs

Approach	Good for	Weak point
Dedicated secrets manager with rotation	Strong, auditable, rotatable secret handling	Real setup/operational investment
Environment variables (no dedicated manager)	Simple, sufficient for low-stakes/early-stage systems	No built-in rotation, less auditable at scale
Hardcoded/baked-in secrets (anti-pattern)	None — never appropriate	Permanently recoverable from history/layers (Part 1.7/7.1)

16. Practical Python/code example

import os
import logging

logger = logging.getLogger("secrets_audit")


def build_downstream_client_never_exposing_secret():
    """
    Constructs a downstream API client using a secret fetched from environment
    configuration (backed by a secrets manager in production, Part 7.1/9.4),
    ensuring the secret is used only in application code — never passed into
    any LLM-bound prompt or context object.
    """
    api_key = os.environ["DOWNSTREAM_API_KEY"]  # never logged, never passed to an LLM call
    return DownstreamServiceClient(api_key=api_key)


def assemble_llm_context(user_message: str, conversation_history: list[dict]) -> list[dict]:
    """
    Assembles LLM context explicitly from only user-facing conversation data —
    deliberately never including application-internal objects (which might
    carry secrets) that a careless implementation could accidentally include.
    """
    return [*conversation_history, {"role": "user", "content": user_message}]

17. Production-quality example

An automated secret-scanning check integrated into CI (Part 7.2), catching accidental secret inclusion in prompt-construction code before it reaches production:

import re
import sys

SECRET_LIKE_PATTERNS = [
    r"sk-[a-zA-Z0-9]{20,}",       # common API key shape
    r"AKIA[0-9A-Z]{16}",          # AWS access key ID shape
]


def scan_file_for_secret_patterns(file_path: str) -> list[str]:
    """
    Scans a source file for secret-shaped patterns, as a CI gate catching
    accidental hardcoding before it reaches a commit or, worse, a prompt.

    Args:
        file_path (str): Path to the file to scan.

    Returns:
        list[str]: Any matched patterns found, empty if none.
    """
    with open(file_path) as f:
        content = f.read()
    return [pattern for pattern in SECRET_LIKE_PATTERNS if re.search(pattern, content)]


if __name__ == "__main__":
    findings = []
    for file_path in sys.argv[1:]:
        matches = scan_file_for_secret_patterns(file_path)
        if matches:
            findings.append((file_path, matches))

    if findings:
        for file_path, matches in findings:
            print(f"POTENTIAL SECRET in {file_path}: {matches}")
        sys.exit(1)  # fails the CI build (Part 7.2), blocking merge

18. Short exercise

A team's incident review finds that a debug log statement, active for two weeks during an investigation, included full request context (which happened to carry an internal service token) in data that was later replayed into subsequent LLM prompts within the same session. Using this chapter's principles, write the specific code-review checklist item that would have caught this before it shipped.

19. Interview questions

• Explain why "instruct the model never to reveal secrets" is an insufficient defense, and what the actual correct mitigation is.
• Why does rate limiting an LLM-backed API endpoint function as both a security control and a cost control simultaneously, in a way that's less true for a typical non-AI API endpoint?
• Describe a code review practice that would catch an accidental secret-to-prompt leakage path before it reaches production.
• Why does a shared, platform-wide provider API key create a cost/blast-radius risk in a multi-tenant AI gateway, and how does per-tenant key scoping address it?
• A large enterprise customer asks to supply their own LLM provider API key rather than using your platform's billing — what changes architecturally, and what stays exactly the same from this chapter's secrets-handling rules?

20. FDE/customer scenario

Customer's security team: "How do you ensure your AI system never accidentally leaks one of our API keys through a chat response?"

The credible answer is structural, not behavioral: secrets are never placed in any code path that constructs an LLM's context in the first place (audited explicitly via code review and automated scanning, section 16/17), so there is nothing present in the model's context for even a successful injection attack (Part 9.1) to disclose — a genuinely stronger guarantee than "we instructed it not to," which this entire chapter (and Part 9.1) has established is not a guarantee at all.

A second, equally common FDE conversation: "We're a large customer — can we use our own OpenAI/Anthropic contract and API key instead of your platform's billing?" Yes, via a BYO-key configuration (section 4): their usage bills to their own provider account and counts against their own negotiated limits, your platform resolves their specific key per request (never a shared platform key for their traffic), and the same secrets-handling discipline (never entering LLM context, stored in a dedicated secrets manager) applies to their key exactly as it would to a platform-managed one — a real, well-understood enterprise pattern, not a special, riskier exception to standard practice.

---

Key takeaways

• A secret must never enter an LLM's context through any code path — the AI-specific extension of the classic "never bake secrets into code/images" principle, since a secret present in context can potentially be disclosed via injection or accidental leakage.
• API security for AI-backed endpoints must weigh attacker-driven LLM cost explicitly, not just traditional data-protection/availability concerns.
• "Instruct the model not to reveal secrets" is not a real defense — the only reliable mitigation is ensuring secrets are structurally never present in context to begin with.
• In a multi-tenant gateway, scope provider API keys per tenant (or honor a BYO-key arrangement) rather than sharing one platform-wide key — a shared key means one tenant's incident degrades or costs every tenant.

Things you should be able to explain

• Why a secret present in an LLM's context is a genuinely different risk than a secret present only in application code.
• Why rate limiting an AI-backed endpoint is simultaneously a security and cost control.
• Why a shared platform-wide provider key is a cost/blast-radius risk in multi-tenant AI systems, and how per-tenant key scoping and BYO-key arrangements address it.

Things you should be able to build

• A prompt-assembly pattern that structurally excludes any path for secrets to enter LLM context, plus a CI-integrated secret-pattern scanner.
• A tenant credential resolver supporting both platform-managed per-tenant keys and customer-supplied BYO keys.

Common mistakes

• Debug logging or overly-broad context objects inadvertently carrying secrets into LLM-bound prompts.
• Relying on prompt instructions rather than structural exclusion to prevent secret disclosure.
• Authentication without rate limiting on LLM-triggering endpoints, missing the cost-attack risk.
• Using one shared provider API key across all tenants in a multi-tenant gateway, with no per-tenant isolation or BYO-key support.

Recommended next chapter

06-tenant-isolation-pii-sandboxing.md