cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › Golden sets

Agent Harness Tutorial · Testing

Golden sets

A golden case is one input with a known right answer. The set is generated from the spec's own limits, so the edges are always covered: a 30-day rule gets cases at 29, 30 and 31. The set is committed to Git, frozen, and never edited to make a failing case pass.

Spec boundaryat_most 30Generated cases29 · 30 · 31Golden JSONLin Git, frozenRun every modelsame harness, cassettesCheapest that passes→ App ConfigurationModelSelector: designed
The last step, the ModelSelector, is designed but not built.

Built (reference agent): 35 cases in evals/golden/eligibility.jsonl, each with id, action, entity, row, expected_allowed, expected_agent_alone and the boundary it tests. A committed baseline (golden pass rate, context size, cost per task $0.000945) may never get worse: a ratchet. The claims agent relies on scenarios instead and has no golden JSONL yet.

Designed: splits (gate, holdout, canary), statuses (active, quarantined, retired), expectations (route, must_call, must_not_call, must_include, judges), growth from about 100 cases per job to 300+ with a frozen holdout. Rules: never edit a case to pass; quarantine a flaky case with a deadline; nobody tunes prompts against the holdout.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

reference-agent/evals/golden/eligibility.jsonl:1The first golden cases: boundaries generated from the spec
{"id": "golden-000", "action": "open_return_request", "entity": "order", "row": {"days_since_delivery": 30, "final_sale": false, "status": "delivered", "total": 0}, "expected_allowed": true, "expected_agent_alone": true, "boundary": "days_since_delivery exactly on its limit"}
{"id": "golden-001", "action": "open_return_request", "entity": "order", "row": {"days_since_delivery": 31, "final_sale": false, "status": "delivered", "total": 0}, "expected_allowed": false, "expected_agent_alone": false, "boundary": "days_since_delivery one past its limit"}
{"id": "golden-002", "action": "issue_refund", "entity": "order", "row": {"days_since_delivery": 0, "final_sale": false, "status": "returned", "total": 10000}, "expected_allowed": true, "expected_agent_alone": true, "boundary": "total exactly on its limit"}
{"id": "golden-003", "action": "issue_refund", "entity": "order", "row": {"days_since_delivery": 0, "final_sale": false, "status": "returned", "total": 10001}, "expected_allowed": true, "expected_agent_alone": false, "boundary": "total one past its limit"}