Eval Harness Tutorial · Writing rules
Presets: start from a ready-made set
A preset is a ready-made set of checks for one kind of agent. Your file names it, then changes it in three ways:
- Change a field: write just that field; the rest of the check stays as the preset has it.
- Remove a check: write the check's name with
null(the clothing agent removes groundedness, because its replies cite tool results, not documents). - Add a check: write a new name, such as your domain patterns.
A preset never ships a judge model: each team must add its own, so client data never goes to a model the client didn't choose.
src/evalharness/presets/customer-support.yaml
# customer-support: version 1's checks for a support agent, run after the fact.
#
# A team's evaluators.yaml names this preset, then adds what only it knows:
# - its judge model (required: judged checks refuse to start without one,
# because for client data the judge must be the client's own model);
# - its domain patterns (tool names and limits; see evalharness.patterns).
# Any check here can be changed field by field, or removed with `null`.
evaluators:
pii:
adapter: presidio_pii
groundedness:
adapter: azure_groundedness
scale: {raw_min: 1, raw_max: 5, higher_is_better: true, threshold: 3}