3.3 — Tool / Function Calling

1. What is it?

Tool calling (also called function calling) is the mechanism by which an LLM, instead of only generating text, can request that a specific function in your code be executed with specific arguments, receive that function's result back, and continue reasoning with it. This is the single mechanism that turns an LLM from "a thing that writes text" into "a thing that can look up real data, take real actions, and interact with real systems" — it is the foundation underneath RAG (Part 3.5), agents (Part 3.7), and MCP (Part 3.8). If you understand this chapter completely, the rest of Part 3 is mostly "more orchestration built on top of this one mechanism," not new fundamentals.

2. Why does it exist?

An LLM's knowledge is frozen at training time (Part 2.6) and entirely "parametric" — baked into its weights, with no way to look anything up. It also has no inherent way to do anything in the world — it can only produce text. Real business problems constantly require both: current data ("what's this customer's account balance right now") and real actions ("create a support ticket," "send this email"). Tool calling exists to bridge exactly this gap, without requiring you to retrain the model every time your data changes or you want it to interact with a new system.

3. What problem does it solve?

It solves "how does a fundamentally text-generating system reliably interact with the deterministic, structured, real world" — databases, APIs, calculators, file systems, other services. Before tool calling was standardized, developers used fragile workarounds: asking the model to output a specially-formatted string that application code would then regex-parse and act on. Tool calling formalizes this into a first-class, schema-validated, provider-supported protocol.

4. How does it work internally?

The full request/response cycle, step by step

This is the part most tutorials gloss over, and it's worth internalizing completely because every agent framework (LangChain, LangGraph, Part 4-5) is, underneath its abstractions, running exactly this loop.

Step 1: Send message + tool definitionseach: name, description, JSON Schema for arguments

Step 2: Model decides whether to call a toolsame next-token generation process (Part 2.4), trained during instruction tuning/alignment (Part 2.6) to recognize a matching tool

Step 3: Model generates tool argumentsstructured-output mechanics (Part 3.2) — typically constrained decoding against the JSON Schema

Step 4: API response returns a "tool use" blockthe model has NOT executed anything — it only requested that you do

Step 5: YOUR code actually executes the real functionthe database query, the API call — and gets a result

Step 6: YOUR code sends the result BACK to the modelas a new "tool result" message, then calls the API again

Step 7: Model generates its next output with the tool's resultfinal answer, or another tool call — this loop can repeat (Part 3.7 agents)

The model never executes anything — it only ever requests, and your code is always the layer in between.

The single most important fact in this entire chapter: the model never executes anything itself. It only ever requests an execution and receives a result back through your code. Every security property, every failure mode, and every architectural decision about tool calling flows from this one fact — you, the application developer, are always the layer between "the model wants to do X" and "X actually happens." This is exactly the control point that Part 9.2 (excessive agency) and Part 9.6 (sandboxing) build their security models around.

Why the model "chooses" to call a tool at all

There's no separate decision-making module bolted onto the LLM for this. The model was trained (via examples during instruction tuning and reinforcement-style alignment, Part 2.6) on data demonstrating exactly this pattern: given a user need and a set of available tool descriptions, produce a tool call when the tool's description matches the need, otherwise produce a normal text response. The quality of your tool's description field is doing real, measurable work here — it's literally the information the model uses to decide relevance, the same way a prompt's wording shapes any other output (Part 3.1). A vague tool description ("does stuff with accounts") produces unreliable tool selection; a precise one ("looks up the current balance and status of a customer account by its account ID; use this whenever the user asks about their current balance") produces reliable selection.

Parallel vs. sequential tool calls

Some providers/models support requesting multiple tool calls in a single turn (e.g., "look up the weather in both Paris and Tokyo" might produce two parallel tool-call requests in one response). Your code is responsible for executing all of them (potentially concurrently — this is exactly where Part 1.1's asyncio.gather pattern applies directly) and returning all their results together before the model continues. Sequential tool calling — where the result of one call is needed before the model can decide on the next — is the more common pattern in multi-step agent reasoning (Part 3.7) and cannot be parallelized, since each step depends on the previous result.

5. Simple mental model

Think of the LLM as a capable analyst sitting behind a glass wall, who can't touch anything themselves. They can see a list of things you're willing to do for them (the tools, with descriptions of what each one does), and they can slide you a request slip specifying exactly what they want done and with what parameters (the tool call). You (your code) are the only one who can actually go do it — check the database, call the API — and you slide the result back through the glass. The analyst then continues their analysis with that new information. They are never, at any point, on your side of the glass.

6. Real-world example

An e-commerce customer support agent needs to answer "where is my order #4521." The LLM alone cannot know this — it's real-time data. With tool calling: the model receives the user's question plus a tool definition get_order_status(order_id: str). It generates a tool call requesting get_order_status(order_id="4521"). Your code executes a real, authenticated, parameterized database query (Part 1.4 — exactly the same SQL-injection-safe query discipline applies here, since the argument came from the model, which itself derived it from user input), gets back {"status": "in_transit", "eta": "2026-09-04"}, and sends that back to the model, which then generates a natural, friendly final answer using the real data. The model never touched the database — your code, with your access controls, did.

7. Architecture diagram

User message"Where is my order #4521?"

LLMsees tool defs: get_order_status

Your application codevalidate args, check RBAC, run real DB query

LLMgenerates final answer using result

The model only ever requests a call — your code decides whether to actually run it.

8. Production considerations

• Treat tool arguments exactly like untrusted user input, because they effectively are — the model derived them from conversation context that ultimately traces back to a user (or, in agentic/RAG systems, to retrieved documents, which is the injection vector in Part 9.1). Validate every argument (type, range, allowed values) before executing anything.
• Enforce authorization at the tool-execution layer, not in the prompt. "Only look up orders belonging to the current user" as a system-prompt instruction is a suggestion the model usually follows; enforcing WHERE user_id = current_user.id in the actual query your code runs is a guarantee. Never rely on the model to self-police access control (Part 9.2, Part 9.4).
• Bound the tool-calling loop. A multi-step agent that calls tools repeatedly needs an explicit maximum number of iterations/tool calls per request — without this, a model stuck in an unproductive loop (or an adversarial input designed to induce one) can run indefinitely, burning cost and potentially taking repeated real-world actions.
• Log every tool call and its result — this is your primary debugging and audit trail for "why did the agent do that," and it's what LangSmith tracing (Part 6.1) is largely built around capturing automatically.
• Design tools to fail informatively, not silently. If a tool call fails (the DB is down, the argument was invalid), return a structured error message to the model rather than raising an unhandled exception that crashes the whole request — this lets the model potentially recover (e.g., ask a clarifying question) instead of the entire interaction failing.

9. Common mistakes

• Giving the model a vague or overly broad tool ("run_query(sql: str)" that accepts arbitrary SQL) instead of narrow, purpose-built tools with constrained parameters — this dramatically increases both unreliable tool selection and security risk (Part 9.2 covers "excessive agency" as exactly this failure pattern).
• Trusting a tool argument because "it came from our own LLM, not directly from the user" — the argument's provenance traces back to user input or retrieved content either way, and should be validated with the same rigor.
• No bound on the tool-calling loop, allowing runaway cost or repeated unintended side effects.
• Poor tool descriptions leading to the model calling the wrong tool, or not calling a tool when it should have — and then blaming "the model" instead of fixing the description, which is often the actual, cheap fix.
• Not handling tool execution failures gracefully, causing the whole conversation to error out instead of degrading (e.g., "I wasn't able to look that up right now").

10. Security considerations

• This is one of the highest-stakes areas in this entire book, covered in full depth in Part 9 (AI Security) — this section previews the core issues, which all stem directly from the "model requests, your code executes" fact in section 4:
  • Excessive agency (Part 9.2): giving a tool more power than the task requires (a "delete_record" tool when only "flag_for_review" was needed) means a manipulated or confused model can cause disproportionate damage.
  • Argument injection: since arguments are model-generated text turned into structured data, any downstream use of those arguments in a shell command, SQL query, or file path must be handled with the same discipline as raw user input (parameterized queries, Part 1.4; no shell string interpolation).
  • Confused deputy risk: a tool executing with the application's credentials rather than the end user's credentials can let a user, through the model, access data or actions they shouldn't personally be authorized for — always scope tool execution to the actual requesting user's permissions, not a broad service credential.

11. Performance considerations

• Each tool-call round trip (model call → your execution → model call again) adds real latency — a multi-step agent that calls 5 tools sequentially pays for 5+ full model round trips, which compounds quickly (Part 3.7, Part 7 cover mitigation strategies like parallelizing independent calls).
• Tool execution itself should be fast and should have its own timeout — a slow tool (an unindexed database query, Part 1.4/1.5) doesn't just slow down that one call, it slows down the entire user-facing interaction, since the model is waiting on it before it can continue.

12. Cost considerations

• Every tool call adds a full round trip's worth of input tokens (the entire conversation history, replayed) and output tokens (the model's next response) — an agent that makes many tool calls per user request can cost substantially more than a single-shot LLM call, a real and often underestimated cost driver (Part 7.10).
• Tool descriptions and schemas are included in every single request to the model (so it knows what's available) — a large number of registered tools with verbose descriptions adds fixed token cost to every call, whether or not that tool is used that turn.

Prompt caching — the direct fix for the replayed-history cost above

The "entire conversation history, replayed" cost above has a direct, mechanical fix: prompt caching. The provider can cache the token-processing state for a stable prefix of your request — your tool definitions, your system prompt, and any conversation turns up to a marked point — so that a subsequent request sharing that exact same prefix reads it back at a fraction of the normal input-token price (roughly a tenth, on current pricing — verify the exact multiplier against current provider documentation) instead of reprocessing it from scratch.

Mechanically, you mark a cache breakpoint — a cache_control field — on a specific content block (the last tool definition, the end of your system prompt, or the last message in a growing conversation). Everything from the start of the request up to that breakpoint becomes the cached prefix; the next request that sends the exact same bytes up to that breakpoint gets a cache hit on that portion and pays full price only for whatever comes after it — in a tool-calling loop, typically just the newest tool result and the model's newest response.

This is precisely why a tool-calling agent loop (section 4) is the canonical use case for caching, not an incidental one: your tool definitions and system prompt are long, byte-identical, and resent on literally every single iteration of the loop — exactly the "large, stable prefix replayed on every call" shape caching is built for. Without caching, a 10-iteration agent loop reprocesses the same multi-thousand-token tool-schema block ten separate times, at full price, purely because it's technically part of a new request each time.

response = await client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=500,
    tools=TOOLS,  # unchanged across loop iterations — the caching target
    system=[
        {
            "type": "text",
            "text": LONG_STABLE_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},  # marks the cache breakpoint
        }
    ],
    messages=messages,
)

Two mechanical facts matter more than the exact syntax, and both are common ways caching silently fails to help: caching is a prefix match — any change anywhere before the breakpoint (a timestamp interpolated into the system prompt, a reordered tool list, non-deterministic JSON serialization of the tool schemas) invalidates the entire cached prefix for that request, silently, with no error, and you're back to paying full price; and a cache hit must be verified explicitly via the response's usage metadata (a cache-read-token field — verify the exact field name against current docs), never assumed just because a cache_control marker is present. Caching doesn't reduce the number of round trips in an agent loop (section 8's iteration cap is still required) — it reduces the cost of each round trip past the first, since the replayed history and tool definitions are read from cache rather than reprocessed at full price.

13. When to use it

Any time an LLM's response needs to depend on real, current, or user-specific data, or needs to trigger a real action in another system — which is the majority of enterprise AI use cases beyond pure open-ended conversation.

14. When NOT to use it

• Purely creative or conversational tasks with no need for external data or action.
• When the "action" is simple, deterministic, and doesn't actually benefit from being mediated by an LLM's judgment — e.g., if a request is always "look up order status by ID with no ambiguity," a plain API endpoint may be simpler and more predictable than routing it through an LLM tool-call decision at all.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Tool/function calling	Flexible, model decides when/what to call based on context	Added latency per round trip; requires careful auth/validation design
Hardcoded logic (no LLM in the loop for the action)	Fully deterministic, fast, simple	Can't handle ambiguous or varied phrasing of the same underlying need
RAG only, no tools (Part 3.5)	Simpler for pure knowledge-lookup, no execution/side-effect risk	Can't take actions or fetch truly real-time/parameterized data

16. Practical Python/code example

from anthropic import AsyncAnthropic
from pydantic import BaseModel

client = AsyncAnthropic()


class OrderStatusArgs(BaseModel):
    """Arguments for looking up an order's current status."""
    order_id: str


async def get_order_status(order_id: str, current_user_id: str) -> dict:
    """
    Looks up an order's status, scoped to the current authenticated user.

    Args:
        order_id (str): The order identifier requested by the model.
        current_user_id (str): The actual authenticated user making the request —
            enforced here, never trusted from the model's context.

    Returns:
        dict: The order's status, or an explicit not-found/unauthorized result.
    """
    order = await db_lookup_order(order_id=order_id, user_id=current_user_id)
    if order is None:
        return {"error": "order not found or not accessible to this user"}
    return {"status": order.status, "eta": str(order.eta)}


TOOLS = [
    {
        "name": "get_order_status",
        "description": (
            "Looks up the current status and estimated delivery date of an order "
            "belonging to the current user, by order ID. Use this whenever the user "
            "asks about the status or location of a specific order."
        ),
        "input_schema": OrderStatusArgs.model_json_schema(),
    }
]


async def handle_support_message(user_message: str, current_user_id: str) -> str:
    """
    Handles one support message, executing at most one tool-call round trip.

    Args:
        user_message (str): The user's message.
        current_user_id (str): The authenticated user's ID, used to scope any tool execution.

    Returns:
        str: The final natural-language response.
    """
    messages = [{"role": "user", "content": user_message}]
    response = await client.messages.create(
        model="claude-sonnet-4-5", max_tokens=500, tools=TOOLS, messages=messages
    )

    tool_use_blocks = [b for b in response.content if b.type == "tool_use"]
    if not tool_use_blocks:
        return response.content[0].text

    tool_results = []
    for block in tool_use_blocks:
        if block.name == "get_order_status":
            args = OrderStatusArgs.model_validate(block.input)
            result = await get_order_status(args.order_id, current_user_id)
            tool_results.append(
                {"type": "tool_result", "tool_use_id": block.id, "content": str(result)}
            )

    messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": tool_results})
    final_response = await client.messages.create(
        model="claude-sonnet-4-5", max_tokens=500, tools=TOOLS, messages=messages
    )
    return final_response.content[0].text

Note current_user_id is a parameter to handle_support_message itself — coming from your authenticated session, never from the model — and is what actually scopes the database query, regardless of what the model puts in its tool call.

17. Production-quality example

Adding a bounded tool-calling loop (supporting multi-step tool use) with logging and a hard iteration cap, directly addressing the "bound the loop" production consideration from section 8:

import logging

logger = logging.getLogger("tool_calling_loop")

TOOL_IMPLEMENTATIONS = {"get_order_status": get_order_status}


async def run_agentic_turn(
    user_message: str, current_user_id: str, max_iterations: int = 5
) -> str:
    """
    Runs a bounded multi-step tool-calling loop for one user turn.

    Args:
        user_message (str): The user's message.
        current_user_id (str): The authenticated user's ID.
        max_iterations (int): Hard cap on tool-call round trips, preventing runaway loops.

    Returns:
        str: The final natural-language response.

    Raises:
        RuntimeError: If the loop exceeds max_iterations without producing a final answer.
    """
    messages = [{"role": "user", "content": user_message}]

    for iteration in range(max_iterations):
        response = await client.messages.create(
            model="claude-sonnet-4-5", max_tokens=500, tools=TOOLS, messages=messages
        )
        tool_use_blocks = [b for b in response.content if b.type == "tool_use"]

        if not tool_use_blocks:
            return response.content[0].text

        messages.append({"role": "assistant", "content": response.content})
        tool_results = []
        for block in tool_use_blocks:
            logger.info("iteration=%d tool=%s args=%s", iteration, block.name, block.input)
            implementation = TOOL_IMPLEMENTATIONS.get(block.name)
            if implementation is None:
                result = {"error": f"unknown tool: {block.name}"}
            else:
                try:
                    result = await implementation(**block.input, current_user_id=current_user_id)
                except Exception as exc:
                    logger.exception("tool execution failed: %s", block.name)
                    result = {"error": f"execution failed: {exc}"}
            tool_results.append(
                {"type": "tool_result", "tool_use_id": block.id, "content": str(result)}
            )
        messages.append({"role": "user", "content": tool_results})

    logger.warning("tool-calling loop exceeded max_iterations=%d", max_iterations)
    raise RuntimeError("exceeded maximum tool-call iterations without a final answer")

Every tool execution is wrapped in its own try/except so one failing tool doesn't crash the whole loop, arguments are validated by unpacking against the implementation's own signature (which should itself validate via Pydantic), and the iteration cap turns an unbounded risk into a bounded, logged, debuggable one.

18. Short exercise

Add a second tool, cancel_order(order_id: str), to the example in section 17. This tool has a real, irreversible side effect. Write out, in plain language, what additional safeguard(s) you would add before allowing this specific tool to execute, beyond what get_order_status (a read-only, side-effect-free tool) needed — and explain why a read tool and a write/destructive tool deserve different levels of caution even though both go through the same mechanism.

19. Interview questions

• Walk through the full tool-calling request/response cycle, being explicit about which steps the model performs and which steps your application code performs.
• Why is it a security mistake to enforce access control only through prompt instructions rather than in the tool-execution code itself?
• Why does a vague tool description cause both reliability problems and, potentially, security problems?

20. FDE/customer scenario

Customer: "We want the assistant to be able to update customer records directly when asked."

Before implementing this, an FDE should walk through the full risk surface this chapter just covered: what specifically triggers the update (only unambiguous, confirmed requests, or any plausible-sounding one)? Is the tool scoped narrowly (e.g., update_customer_phone_number with validated format) rather than broadly (update_customer_record(field: str, value: str) accepting arbitrary fields)? Is there a human-in-the-loop confirmation step for anything destructive or hard-to-reverse (Part 5.4, Part 8.5)? Is execution scoped to records the requesting user is actually authorized to modify? This is the concrete, technical version of the FDE instinct from Part 12: don't just build what was asked for literally — understand the actual risk and constrain the solution to match it.

---

Key takeaways

• The model only ever requests a tool call; your application code is always the layer that actually executes it — every security and reliability property flows from this fact.
• Tool descriptions and argument schemas are doing real, measurable work in shaping model behavior — treat them with the same care as prompts (because they are prompts, structurally).
• Authorization must be enforced in tool-execution code, never assumed from prompt instructions alone.

Things you should be able to explain

• The full seven-step tool-calling request/response cycle.
• Why tool arguments must be validated with the same rigor as raw user input.

Things you should be able to build

• A bounded, logged, multi-step tool-calling loop with per-tool error handling and user-scoped authorization.

Common mistakes

• Overly broad tools ("run arbitrary SQL") instead of narrow, purpose-built ones.
• No iteration cap on multi-step tool-calling loops.
• Enforcing access control only in the prompt instead of in code.

Recommended next chapter

04-vector-databases.md