D5 · Context & Reliability › Cluster 8 · Observability & Production Ops

8.5 · Safety & Guardrails at Runtime

Bound what the agent can do

CONCEPT Guardrails wrap the model on both sides

Evals (8.4) catch problems before you ship; runtime guardrails catch them as they happen. Think of a guardrail sandwich: validate the input going in, constrain the model, then validate the output coming out — before it reaches a user or triggers a side effect. Neither side alone is enough: input checks stop obvious abuse, output checks catch what the model generated anyway.

StageGuardrailExample
InputValidation + moderation passReject oversized/malformed input; classify prompts for policy before they reach the model
ModelGround & constrainAllow "I don't know"; restrict to provided context; require citations
OutputValidation + classification passSchema/format check; a moderation model screens the response before display
ActionsAllow-list tools · spend/rate caps · human-in-the-loopOnly whitelisted tools; per-tenant token cap; confirm irreversible actions
🔑 Reduce hallucinations at the model layer: explicitly let Claude say "I don't know," restrict it to provided documents (not general knowledge), and have it cite quotes so each claim is auditable and unsupported ones get retracted.

CONCEPT Bound the blast radius of actions

An agent that can call tools can cause real-world harm, so bound what it's allowed to do. Allow-list tools — deny-by-default, expose only the tools this agent needs. Impose spend and rate caps so a runaway loop can't burn the budget or blow past your OTPM limit (8.3). And keep a human in the loop for risky, irreversible actions — the model proposes, a person approves before send_email, delete, or a payment executes. These are the same reversibility-and-gating instincts from D1, applied as production controls.

Deny-by-default everywhere: unknown tool → refuse; missing schema field → reject; spend over cap → halt. A guardrail that fails open isn't a guardrail.

DIAGRAM The guardrail sandwich

flowchart TD IN["User / upstream input"] --> IV{"Input valid
+ passes moderation?"} IV -->|"no"| REJ["🔴 Reject early"] IV -->|"yes"| M["Model call
grounded · may say 'I don't know'"] M --> OV{"Output valid
+ passes classifier?"} OV -->|"no"| BLOCK["🔴 Block / retract claim"] OV -->|"yes"| ACT{"Action requested?"} ACT -->|"tool not allow-listed
or over spend cap"| DENY["🔴 Deny"] ACT -->|"risky / irreversible"| HITL["🟠 Human-in-the-loop
confirm"] ACT -->|"safe + within caps"| RUN["🟢 Execute"] HITL --> RUN classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class RUN ok; class HITL gate; class REJ bad; class BLOCK bad; class DENY bad;

CODE Input check → model → output check → gated action

Deny-by-default at every seam: moderate input, validate output, allow-list the tool, enforce a spend cap, and gate irreversible actions behind a human.

ALLOWED_TOOLS = {"search", "read_file"}          # deny-by-default allow-list
IRREVERSIBLE  = {"send_email", "delete_file", "charge_card"}

def handle(user_input):
    if not moderate(user_input):                  # input moderation pass
        return "Refused: input violates policy."
    if spend.today > SPEND_CAP:                    # spend/rate cap
        return "Halted: daily budget reached."

    resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024,
              system=GROUNDED_SYSTEM,               # "use only the docs; you may say I don't know"
              messages=[{"role": "user", "content": user_input}])

    if not validate_output(resp) or not classify_safe(resp):
        return "Blocked: output failed validation."   # output guardrail

    for tool in tool_calls(resp):
        if tool.name not in ALLOWED_TOOLS:
            return f"Denied: '{tool.name}' not allow-listed."
        if tool.name in IRREVERSIBLE and not human_approves(tool):
            return f"Denied: '{tool.name}' needs human approval."  # HITL
        run_tool(tool)

TRADE-OFFS Guardrail coverage vs. latency & friction

✅ Layered guardrails

  • Input + output classification catches abuse both ways
  • Allow-lists + caps bound the blast radius
  • HITL stops irreversible mistakes cold

⚠️ The cost of layering

  • Extra moderation calls add latency & token cost
  • HITL on everything defeats autonomy — gate by risk
  • Over-tight input filters reject legitimate users

EXAM TRAPS

Trap: Validating input but not output. → The model can still emit unsafe or malformed content. Guardrail both sides — an output classification/validation pass before anything is shown or acted on.
Trap: Gating every action behind a human. → That kills the agent's value. Gate by risk / reversibility — auto-run safe reversible tools, HITL only the irreversible ones.
Trap: No spend or rate cap. → A looping agent can exhaust your budget or OTPM (8.3) in minutes. Caps that fail closed are a mandatory runtime control.
Trap: Relying on the prompt alone to prevent hallucination. → Prompt grounding (allow "I don't know", restrict to provided docs, cite quotes) helps but doesn't eliminate it — pair it with an output verification/validation layer for high-stakes claims.

PRACTICE

Q1 · Why validate output when you already validate input?

Input checks stop bad requests, but the model can still generate unsafe, hallucinated, or malformed output from a perfectly benign prompt. A separate output validation / classification pass screens the response before it reaches a user or triggers an action — you need both sides of the sandwich.

Q2 · Three runtime controls that bound what an agent's tool calls can do?

Allow-list the tools (deny-by-default, expose only what's needed); spend/rate caps so a runaway loop can't drain budget or blow the OTPM limit; and human-in-the-loop approval for risky, irreversible actions before they execute.

Q3 · Which prompt-level techniques reduce hallucination at runtime?

Explicitly allow the model to say "I don't know," restrict it to the provided documents rather than general knowledge, and require it to cite quotes so each claim is auditable and unsupported ones are retracted. Back these with an output-verification layer for high-stakes use.