Eval Harness Tutorial · Architecture
The second engine: cost rules
The five patterns judge one conversation. Most cost waste can't be seen in one conversation: “a nightly job paying real-time prices” or “a cache that expires before it's reused” only shows across a whole day of calls. So cost rules run on a second engine: llm-cost-explainer, our own product, used here as a library. It reads a provider's usage export and runs its rules (YAML plus SQL on DuckDB).
Our adapter turns each of its rules into an ordinary verdict, so cost goes to the same App Insights events and the same weekly report as everything else.
| Rule outcome | Verdict | Why |
|---|---|---|
| Found waste | fail, one finding per API key or model, with the monthly saving | So the report can rank by money |
| Ran, found nothing | pass | |
| The export lacks what the rule needs | skip: could not judge | An export with no hourly data can't show a nightly job; that's never a pass |
Run when this page was built
llm-cost-explainer's own made-up days, each with a known mistake, through our adapter:
A day with a nightly job sent through the real-time API
| Rule | What it finds | Verdict | Why |
|---|---|---|---|
LLM-01 | Long prompts sent with caching off | fail | 1 finding, about $614/month to save; 30,332,921 input tokens went to gpt-4.1 on key key_etl today, and 0% of them were read from the cache. |
LLM-06 | Bulk work on the real-time API | fail | 1 finding, about $1,341/month to save; Key key_etl sends $89.38/day of real-time traffic to gpt-4.1 that runs in one burst of a few hours (3 active hours). Peop |
LLM-08 | — | pass | nothing wasteful found |
LLM-22 | — | pass | nothing wasteful found |
LLM-24 | — | pass | nothing wasteful found |
LLM-26 | — | pass | nothing wasteful found |
A day with a model that retires next month still in use
| Rule | What it finds | Verdict | Why |
|---|---|---|---|
LLM-01 | — | pass | nothing wasteful found |
LLM-06 | — | pass | nothing wasteful found |
LLM-08 | — | pass | nothing wasteful found |
LLM-22 | — | pass | nothing wasteful found |
LLM-24 | Old model still in use | fail | 1 finding, about $773/month to save; claude-sonnet-4-5 retires on 2026-11-30 and cost $77.34 today (keys: key_legacy). |
LLM-26 | — | pass | nothing wasteful found |
6 of 36 rules are built (LLM-01, LLM-06, LLM-08, LLM-22, LLM-24, LLM-26); every one now runs through this framework. The other 30 come next in Tier 5, in an order you choose. The full list is on checks for any agent.
src/evalharness/costs.py:62def results_of(payload: dict[str, Any]) -> list[EvalResult]:
"""One result per rule, from llm-cost-explainer's answer."""
version = _version()
found: dict[str, list[dict[str, Any]]] = {}
for f in payload.get("findings", ()):
found.setdefault(f["rule_id"], []).append(f)
titles = {f["rule_id"]: f["title"] for f in payload.get("findings", ())}
out = []
for rule_id in sorted(set(payload.get("rules_run", ())) | set(found)):
hits = found.get(rule_id, [])
saving = sum(f["savings_day"] for f in hits)
out.append(
EvalResult(
verdict="fail" if hits else "pass",
score=0.0 if hits else 1.0,
raw_score=saving,
threshold=None,
reason=(
f"{len(hits)} finding{'s' if len(hits) != 1 else ''}, "
f"about ${saving * 30:,.0f}/month to save; {hits[0]['summary']}"
if hits
else "nothing wasteful found"
),
evaluator=rule_id,
version=version,
provider=PROVIDER,
label=titles.get(rule_id, ""),
raw={"day": payload.get("day"), "findings": hits},
findings=tuple(
Finding(
kind=rule_id,
confidence=1.0,
label=f"{', '.join(f'{k}={v}' for k, v in f['scope'].items())}: "
f"${f['savings_day'] * 30:,.0f}/month",
)
for f in hits
),
)
)
for rule_id, title, why in payload.get("skipped", ()):
if rule_id in found or any(r.evaluator == rule_id for r in out):
continue
out.append(
EvalResult(
verdict="skip",
score=None,
raw_score=None,
threshold=None,
reason=f"could not judge: {why}",
evaluator=rule_id,
version=version,
provider=PROVIDER,
label=title,
)
)
return sorted(out, key=lambda r: r.evaluator)- Which cost rules matter most to your clients? This sets the order of the remaining 30.
Reply in chat with the page name and the number, for example “session 2: …”.