cleandataengineering.aiEval Harness, drawn simply
You are here: Writing rules › Presets

Eval Harness Tutorial · Writing rules

Presets: start from a ready-made set

A preset is a ready-made set of checks for one kind of agent. Your file names it, then changes it in three ways:

  • Change a field: write just that field; the rest of the check stays as the preset has it.
  • Remove a check: write the check's name with null (the clothing agent removes groundedness, because its replies cite tool results, not documents).
  • Add a check: write a new name, such as your domain patterns.

A preset never ships a judge model: each team must add its own, so client data never goes to a model the client didn't choose.

src/evalharness/presets/customer-support.yaml
# customer-support: version 1's checks for a support agent, run after the fact.
#
# A team's evaluators.yaml names this preset, then adds what only it knows:
#   - its judge model (required: judged checks refuse to start without one,
#     because for client data the judge must be the client's own model);
#   - its domain patterns (tool names and limits; see evalharness.patterns).
# Any check here can be changed field by field, or removed with `null`.
evaluators:
  pii:
    adapter: presidio_pii
  groundedness:
    adapter: azure_groundedness
    scale: {raw_min: 1, raw_max: 5, higher_is_better: true, threshold: 3}