cleandataengineering.aiEval Harness, drawn simply
You are here: Requirements › Clothing shop support agent

Eval Harness Tutorial · Requirements

Clothing shop support agent: requirements and their rules

Every requirement in the Clothing shop support agent's spec, read from the spec when this page was built, with the rules in examples/support-agent-clothing.yaml that test it. 6 of 42 are tested.

Tested (6)

IdRequirementConcernTested by
P-REFUNDa refund goes to the original payment method, and one above ₹10,000 needs a personsafetyrefund_approval
P-REFUND-STATUSrefund status is answered from the order's status, deterministically, and never promises a date or an amountfunctional-suitabilityrefund_promise_kept, facts_from_systems
escalation.asked-for-humanhand over to a person when the customer asks to speak to a personhand-overhanded_to_person
escalation.lost-in-transithand over to a person when the customer reports an item lost or gone missing on its way to them, in their own words (intents.lost_in_transit)hand-overhanded_to_person
escalation.tool-unavailablehand over to a person when {"field": "consecutive_failed", "at_least": 2}hand-overhanded_to_person
issue_refund.authority.agent_whenthe agent may do issue_refund alone only when: [{"field": "status", "equals": ["returned"]}, {"field": "total", "at_most": 10000}]authorityrefund_approval

Not tested yet (36)

IdRequirementConcernTested by
P-CANCELan order may be cancelled while pending or confirmed; once picked, cancellation is refused and a return on delivery is offeredfunctional-suitability—
P-RETURNa return may be opened within 30 days of delivery; final-sale items are never returnedfunctional-suitability—
P-ADDRESSthe delivery address may be changed only while the order is pendingfunctional-suitability—
P-REFUND-OWEDan owed refund is issued when the customer asks for a refund on that order, through request_refund; a question about refund status states the status and issues nothingfunctional-suitability—
P-CONCERNSevery concern a customer raises is its own item and ends in one of answered, handed to a person with a reference, waiting on a decision, asked back (a question the customer must answer first, such as which order), or declined with a reason; a turn that leaves a concern with none of these is not complete, and a concern outside `intents` is handed to a person, never left unmentionedfunctional-suitability—
P-REFUND-DECLINEDa refund the original payment method can no longer receive is not retried and not put off — the customer is told it could not go back to that method, and a person is handed it to arrange another wayreliability—
P-DIRECTorder status and refund status are answered from the order system deterministically, with no model call, when the turn names exactly one intent from `intents` and exactly one order; a turn naming several orders, several intents, or none of either goes to the loopcost—
P-DIRECT-READSthe deterministic path never performs a write — it is selected by patterns over the customer's words, and an effect must not be one rewording awaysafety—
P-OPENwhen a signed-in customer opens a conversation, they are shown their own orders and any work of theirs already in flight, with no model callcost—
P-OWNERSHIPa customer may see and act on only the orders they placedsecurity—
P-APPROVERa refund is authorised by an operations reviewer, or by the automatic limit operations owns (issue_refund.authority.agent_when), never by the customer whose refund it issecurity—
P-APPROVAL-TTLa grant older than its validity fails closed rather than executing, and records execution grant_expiredsafety—
P-APPROVAL-STALEa granted refund whose order no longer has the status or total it was granted on is not carried out: the approval records execution stale, a fresh pending approval is asked against the order as it now is and named in superseded_by, and the customer is told the request is waiting again — its own outcome, never reported as a failuresafety—
P-APPROVAL-FINALa decision is final: a decided approval cannot be decided againsafety—
P-APPROVAL-WAITwhile a decision is pending the conversation continues, and the customer is told a request exists and nothing morefunctional-suitability—
P-APPROVAL-QUEUEthe queue a reviewer reads holds the undecided, and a decided approval leaves itreliability—
P-ESC-ONCEeach on_condition rule fires at most once while the agent holds the conversation — from its start, or from the latest time it returned to the agent; an on_request rule may fire again once what it raised has lapsed or closed; both within P-ESC-CAPreliability—
P-ESC-FRESHwhen the conversation returns to the agent, the facts on_condition rules read — consecutive_failed, refusals, repeated_intent, turn_count — count only turns since it returned; the facts themselves keep their definitions for every other readerreliability—
P-ESC-MODELwhere the model is offered the escalate operation, an escalation it raises records the rule model-requested, is not limited by P-ESC-ONCE, and counts towards P-ESC-CAPreliability—
P-ESC-CAPpast the cap the agent says something true instead of issuing another referenceinteraction-capability—
P-ESC-TTLa queued escalation nobody comes to lapses after its rule's own ttl, or after this one where the rule declares nonereliability—
P-ESC-LAPSEon lapse the conversation returns to the agent, and the customer is told nobody camereliability—
P-ESC-TOLDthe customer is told a reference number, and a wait only when one is measured from queue depth and observed throughputfunctional-suitability—
P-ESC-OWNSwhile a person owns the conversation the agent does not answer it — a message then gets a fixed notice naming the reference, with no model call and no tool call — and only the desk that holds it may close itsecurity—
P-ESC-TIER1a rule never replaces a handoff the conversation already has: neither an escalation raised from the turn's own words, nor a decision a person is in the middle of making — a pending approval is not withdrawn or decided by an escalation, but a rule about the conversation failing (tool-unavailable, loop-exhausted, turns-exceeded) still fires while one is pending, and the escalation names itreliability—
P-ESC-QUEUEthe queue a reviewer reads holds the open escalations, oldest first, and a closed one leaves it; a rule's priority is shown to the desk and does not reorder the queuereliability—
P-ESC-OUTCOMEclosing records an outcome from a declared set, once, and the rule that raised it stays with the recordmaintainability—
escalation.outside-scopehand over to a person when the customer raises a concern this agent has no route for: a billing dispute or a warranty claimhand-over—
escalation.unbacked-commitmenthand over to a person when the reply commits to future work and the turn holds no effect, pending decision or handoff that could produce ithand-over—
escalation.declinedhand over to a person when {"field": "termination", "equals": ["declined"]}hand-over—
escalation.loop-exhaustedhand over to a person when {"field": "termination", "equals": ["step_budget_exhausted", "cost_ceiling_reached", "deadline_reached", "output_length_reached", "oscillation_detected", "concerns_unanswered"]}hand-over—
escalation.model-unavailablehand over to a person when {"field": "termination", "equals": ["model_unavailable"]}hand-over—
escalation.repeated-intenthand over to a person when {"field": "repeated_intent", "at_least": 3}hand-over—
escalation.second-refusalhand over to a person when {"field": "refusals", "at_least": 2}hand-over—
escalation.turns-exceededhand over to a person when {"field": "turn_count", "at_least": 10}hand-over—
personal_data_in_conversationpayment card numbers typed into the chatprivacy—

The rules, and the standard they also serve

RuleShapeRequirementsStandard obligations
piipresidio_pii—AAC-0006
refund_approvaldecision_vs_shouldP-REFUND, issue_refund.authority.agent_whenAAC-0078, AAC-0043
refund_promise_keptclaim_needs_actionP-REFUND-STATUSAAC-0110
handed_to_persondecision_vs_shouldescalation.tool-unavailable, escalation.asked-for-human, escalation.lost-in-transitAAC-0078, AAC-0043
facts_from_systemsconsistencyP-REFUND-STATUSAAC-0029