Learn › Agent Harness, the big picture › Stop 7 › What a harness must do
Stop 7 of 9 · Pages
Prove and change: evals, tests and release
How you know the agent is good, and how you improve it without breaking it.
- 25 · Evals
- 26 · Attack it before others do
- 27 · Release safely
- 28 · Change settings without changing code
- 29 · Learn from every miss
Evals
Also called: Evaluations, golden sets, graders
Is it any good, and how would we know?
Why
The same input can give a different output each time, so ordinary pass-or-fail tests are not enough. An eval runs many cases and reports how many came out right.
Without it
"It seemed fine when I tried it" is the only evidence.
Think about
- A set of real cases with known good outcomes
- How each is graded: a rule in code, a model as judge, or a person
- When it runs: before a release, and on live traffic afterwards
In the big picture: Eval library, in the Change family
Attack it before others do
Also called: Red teaming, adversarial testing
Does the harness hold when someone tries to break it?
Why
Attackers will try hidden orders, false identities and persuasion. It is cheaper to try them first.
Without it
The first test of the guardrails is a real attack.
Think about
- A library of known attacks, run before every release
- People whose job is to find new ones
- Every hole found becomes a permanent test
In the big picture: Eval library, and the Lifecycle rail
Release safely
Also called: Versioning, staged rollout, rollback
How does a change reach users without hurting them?
Why
Changing one sentence in a prompt, or moving to a newer model, can change behaviour everywhere. Each change needs a version and a way back.
Without it
Every improvement is a gamble on live customers.
Think about
- A version for prompts, models and tools, not only for code
- The evals as a gate that every change must pass
- A small group first, and a quick way back
In the big picture: Release and config, in the Change family
Change settings without changing code
Also called: Configuration
Can we change a limit without a new release?
Why
Limits, prompts and model choices change far more often than code. Keeping them as settings makes each change small, visible and reversible.
Without it
A one-line change to a limit waits two weeks for a release.
Think about
- Which things are settings: limits, prompts, models, lists of allowed tools
- Who may change each one
- A record of every change, because settings are powerful
In the big picture: Release and config, in the Change family
Learn from every miss
Also called: Feedback loop
How does a mistake in production make the agent better?
Why
Production finds the cases nobody imagined. Each one is worth keeping, because it is a real question with a known right outcome.
Without it
The same failure returns after the next change.
Think about
- How misses are noticed: complaints, thumbs down, alerts, reviews
- Who turns a miss into a case
- How the set of cases is kept healthy as it grows
In the big picture: The Lifecycle rail, and the Eval library