cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › In production

Agent Harness Tutorial · Testing

Testing in production

Tests before release can't see everything. In production, a synthetic customer keeps asking known questions, new releases take a small slice of traffic first, and every real miss becomes a test case.

PracticeWhat it doesStatus
Synthetic canarya fake customer asks known questions through the real front door every 10 minutes; its turns are marked synthetic and left out of ratesbuilt in the reference agent; planned for Azure
Canary releaseContainer Apps revisions split traffic 95/5 to a new releaseinfra ready; not run yet
Shadow moderun a new release on real traffic without answering, and comparedesigned
Online rulesre-check finished turns and raise findingsonline eval
Miss → test casea production failure becomes a golden case or scenarioaccepted gap, review 1 Nov 2026

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

"""The canary: a synthetic customer through the deployed edge, on a schedule (T-056).

    uv run python scripts/canary.py --every 600          # forever
    uv run python scripts/canary.py --once

Signs in as C-7001 — a realm user whose orders nobody else has — through the
same public client a customer's chat uses, sends the four cases in
`support_agent.watch.canary` to `/chat`, and counts each as passed or failed.
Prometheus pages on a failure and on silence (deploy/prometheus/rules/agent.yml).

With no realm configured (`AGENT_ISSUER_URL` empty) the token comes from the
local test issuer, which only a server started without a realm accepts.
"""

from __future__ import annotations

import argparse
import os
import sys
import time

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from evals import issuer as local_issuer  # noqa: E402

from agent_harness import telemetry as tel  # noqa: E402
from support_agent.config import Settings  # noqa: E402
from support_agent.watch import canary  # noqa: E402

CUSTOMER = "C-7001"


def token_source(settings: Settings):
    if not settings.issuer_url: