cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › Cost rules

Eval Harness Tutorial · Architecture

The second engine: cost rules

The five patterns judge one conversation. Most cost waste can't be seen in one conversation: “a nightly job paying real-time prices” or “a cache that expires before it's reused” only shows across a whole day of calls. So cost rules run on a second engine: llm-cost-explainer, our own product, used here as a library. It reads a provider's usage export and runs its rules (YAML plus SQL on DuckDB).

Our adapter turns each of its rules into an ordinary verdict, so cost goes to the same App Insights events and the same weekly report as everything else.

Usage exportOpenAI or Anthropic,a day of callsllm-cost-explainerits YAML + SQL ruleson DuckDB (a library,still its own product)LlmCostExplainerone verdict per ruleSame sinks,same weeklyreport
A day of calls in, one verdict per rule out, into the same places as every other check.
Rule outcomeVerdictWhy
Found wastefail, one finding per API key or model, with the monthly savingSo the report can rank by money
Ran, found nothingpass
The export lacks what the rule needsskip: could not judgeAn export with no hourly data can't show a nightly job; that's never a pass

Run when this page was built

llm-cost-explainer's own made-up days, each with a known mistake, through our adapter:

A day with a nightly job sent through the real-time API

RuleWhat it findsVerdictWhy
LLM-01Long prompts sent with caching offfail1 finding, about $614/month to save; 30,332,921 input tokens went to gpt-4.1 on key key_etl today, and 0% of them were read from the cache.
LLM-06Bulk work on the real-time APIfail1 finding, about $1,341/month to save; Key key_etl sends $89.38/day of real-time traffic to gpt-4.1 that runs in one burst of a few hours (3 active hours). Peop
LLM-08—passnothing wasteful found
LLM-22—passnothing wasteful found
LLM-24—passnothing wasteful found
LLM-26—passnothing wasteful found

A day with a model that retires next month still in use

RuleWhat it findsVerdictWhy
LLM-01—passnothing wasteful found
LLM-06—passnothing wasteful found
LLM-08—passnothing wasteful found
LLM-22—passnothing wasteful found
LLM-24Old model still in usefail1 finding, about $773/month to save; claude-sonnet-4-5 retires on 2026-11-30 and cost $77.34 today (keys: key_legacy).
LLM-26—passnothing wasteful found
How far the cost pack is

6 of 36 rules are built (LLM-01, LLM-06, LLM-08, LLM-22, LLM-24, LLM-26); every one now runs through this framework. The other 30 come next in Tier 5, in an order you choose. The full list is on checks for any agent.

Each rule as a verdict, in real codesrc/evalharness/costs.py:62
def results_of(payload: dict[str, Any]) -> list[EvalResult]:
    """One result per rule, from llm-cost-explainer's answer."""
    version = _version()
    found: dict[str, list[dict[str, Any]]] = {}
    for f in payload.get("findings", ()):
        found.setdefault(f["rule_id"], []).append(f)
    titles = {f["rule_id"]: f["title"] for f in payload.get("findings", ())}
    out = []
    for rule_id in sorted(set(payload.get("rules_run", ())) | set(found)):
        hits = found.get(rule_id, [])
        saving = sum(f["savings_day"] for f in hits)
        out.append(
            EvalResult(
                verdict="fail" if hits else "pass",
                score=0.0 if hits else 1.0,
                raw_score=saving,
                threshold=None,
                reason=(
                    f"{len(hits)} finding{'s' if len(hits) != 1 else ''}, "
                    f"about ${saving * 30:,.0f}/month to save; {hits[0]['summary']}"
                    if hits
                    else "nothing wasteful found"
                ),
                evaluator=rule_id,
                version=version,
                provider=PROVIDER,
                label=titles.get(rule_id, ""),
                raw={"day": payload.get("day"), "findings": hits},
                findings=tuple(
                    Finding(
                        kind=rule_id,
                        confidence=1.0,
                        label=f"{', '.join(f'{k}={v}' for k, v in f['scope'].items())}: "
                        f"${f['savings_day'] * 30:,.0f}/month",
                    )
                    for f in hits
                ),
            )
        )
    for rule_id, title, why in payload.get("skipped", ()):
        if rule_id in found or any(r.evaluator == rule_id for r in out):
            continue
        out.append(
            EvalResult(
                verdict="skip",
                score=None,
                raw_score=None,
                threshold=None,
                reason=f"could not judge: {why}",
                evaluator=rule_id,
                version=version,
                provider=PROVIDER,
                label=title,
            )
        )
    return sorted(out, key=lambda r: r.evaluator)
Your review · Cost rules
  1. Which cost rules matter most to your clients? This sets the order of the remaining 30.

Reply in chat with the page name and the number, for example “session 2: …”.