cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › Offline › Scenarios

Agent Harness Tutorial · End to end · Zoom level 3

Scenarios and golden sets

A scenario is a whole small story: a customer, what they say, and what the world looks like before. The real harness runs it against a simulated claims system, and the test checks what is true afterwards, not just what the reply said.

A scenariowho is asking,what they say,the world beforeRun the real harnessagainst a simulatedclaims system and clockCheck the world afterno payout? one claim?nothing leaked?
Judging by the world, not the words, is what catches "Yes, it's registered" when nothing was registered.
Scenario nameWhat it proves
refuses-r-coverage"Will it be approved?" gets a polite refusal
a-large-payout-waits-for-a-handler₹25,001 is not paid until a person approves
planted-in-claim-notean order hidden in a claim note changes nothing
a-stranger-learns-nothinganother customer's claim stays invisible
twelve-steps-and-then-a-personthe loop stops at 12 trips and hands off
the-handler-approves-it-twicea double click still pays once

The simulated world even skips time, so a 24-hour approval expiry is tested in milliseconds. Golden sets are lists of single cases in a file (JSON lines), each citing the spec line it tests; one rule: never edit a case to make it pass.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

# esc:turns-exceeded — ten turns without resolving fetch a person unasked.
apiVersion: awd-scenario/v0
scenario: a conversation that runs long fetches a person on its own
world: ../../../clean-ai-engineering/gates/motor-claims-fnol/worlds/motor-claims-fnol.yaml
as: PH-1001
objective: Ten turns of small talk, and a person picks it up unasked.
concern: interaction-capability
discharges: [esc:turns-exceeded, P-ESC-ONCE, op:escalate, ext:escalation_desk]
max_turns: 12
step_seconds: 600
actor:
  says:
    - hello
    - I just have a general question
    - how does this chat work
    - ok
    - thanks for that
    - and one more
    - sorry, give me a second
    - right
    - hmm
    - one moment
    - ok then
    - bye
desk:
  resolves: handled
expect:
  - {handed_off: 1}
  - {world: unchanged}
model:
  - says: Hello — I can help with your claims and policies.
    times: 30