Agent Harness Tutorial · Testing
Testing in production
Tests before release can't see everything. In production, a synthetic customer keeps asking known questions, new releases take a small slice of traffic first, and every real miss becomes a test case.
| Practice | What it does | Status |
|---|---|---|
| Synthetic canary | a fake customer asks known questions through the real front door every 10 minutes; its turns are marked synthetic and left out of rates | built in the reference agent; planned for Azure |
| Canary release | Container Apps revisions split traffic 95/5 to a new release | infra ready; not run yet |
| Shadow mode | run a new release on real traffic without answering, and compare | designed |
| Online rules | re-check finished turns and raise findings | online eval |
| Miss → test case | a production failure becomes a golden case or scenario | accepted gap, review 1 Nov 2026 |
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
reference-agent/scripts/canary.py:1The synthetic canary"""The canary: a synthetic customer through the deployed edge, on a schedule (T-056).
uv run python scripts/canary.py --every 600 # forever
uv run python scripts/canary.py --once
Signs in as C-7001 — a realm user whose orders nobody else has — through the
same public client a customer's chat uses, sends the four cases in
`support_agent.watch.canary` to `/chat`, and counts each as passed or failed.
Prometheus pages on a failure and on silence (deploy/prometheus/rules/agent.yml).
With no realm configured (`AGENT_ISSUER_URL` empty) the token comes from the
local test issuer, which only a server started without a realm accepts.
"""
from __future__ import annotations
import argparse
import os
import sys
import time
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from evals import issuer as local_issuer # noqa: E402
from agent_harness import telemetry as tel # noqa: E402
from support_agent.config import Settings # noqa: E402
from support_agent.watch import canary # noqa: E402
CUSTOMER = "C-7001"
def token_source(settings: Settings):
if not settings.issuer_url: