Eval Harness Framework
One framework to check AI agents with any provider's evals, guardrails, cost and business rules.
1 · Purpose
The Eval Harness Framework checks AI agents for quality, security, cost and a business's own rules. It mixes checks from several providers in one run and gives one result format. It is for teams running agents who already use tools such as LangSmith, Langfuse or Application Insights and are tired of writing glue between them. It brings no database, UI or dashboards of its own: it reads from and writes back to the tools a team already has.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Add PAN and Aadhaar recognisers to the Presidio adapter
- Write contract test rows for one adapter
- Review the claims example YAML as a newcomer
4 · Requirements
What it must do
- Runs checks from Azure AI Evaluation, DeepEval, Ragas and Presidio through one request and result shape.
- Loads which checks run, and where, from a YAML file a team can read on one screen.
- Ships a customer-support preset that a team's file overrides field by field.
- Checks whole conversations with five domain patterns, such as an action that needs another action first.
- Lets a free hosted model act as judge, so no paid cloud account is needed.
- Is already used by the support agent and the claims agent.
How well it must do it
- No vendor SDK is imported outside its own adapter.
- Every adapter must pass the same contract test table for its port.
- Deterministic code checks come before judge models.
- Version 1 must install with one command and produce a first report within an hour.
- A feature is added only when a real team asks for it, or it replaces something.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS | GCP |
|---|---|---|---|---|
| Trace source | OpenAI-format messages ✓; Phoenix, Langfuse (planned) | Application Insights (planned) | — | — |
| Plan | YAML ✓ | — | — | — |
| Evaluator adapters | DeepEval, Ragas, Presidio ✓ | Azure AI Evaluation groundedness ✓; Content Safety (planned) | Bedrock evaluations and guardrails (planned) | planned, no work started |
| Judge | Groq-hosted open model ✓ | — | — | — |
| Result sink | OpenTelemetry evaluation events (planned) | Application Insights custom events (planned) | — | — |
7 · What needs to be done
Tier 0: spike 2 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Run four providers through one request shape | done | M | Python |
| Record every provider's real answer for offline tests | done | S | Testing |
Tier 1: core package 3 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Check the competitor landscape with sources | done | M | Research |
| Extract the package and use it from both reference agents | done | M | Python packaging |
| Contract test table per port | done | S | Testing |
| A stability label on every adapter | todo | S | Python |
Tier 2: sessions and domain patterns 2 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Session shape and five domain patterns | done | M | Python |
| Worked example for the claims agent | done | S | YAML |
| Run the five patterns on real claims and support conversations | todo | M | AI evaluation |
Tiers 3 to 6: version 1 slice 0 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Read traces from Phoenix and Application Insights | todo | M | Observability |
| Write results back as OpenTelemetry evaluation events | todo | M | OpenTelemetry |
| Bring in the six built LLM cost rules | todo | M | Cost engineering |
| Add Indian ID formats (PAN, Aadhaar) to the personal-data check | todo | S | Security |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Adapter developer | 2 | 3-5 hours/week | Adds and maintains one provider's adapter and its contract tests. |
| Rule author | 1 | 2-3 hours/week | Writes domain patterns and cost rules in YAML. |
| Pilot team contact | 1 | 1-2 hours/week | Runs version 1 on a real team's traces and reports what they found. |
9 · What you'll learn
- How the main eval and guardrail tools differ in their data shapes.
- How ports and adapters keep vendor code in one place.
- Why checking and deciding are separate steps.
- How to write a business rule that checks a whole conversation.
- How OpenTelemetry carries evaluation results.
Level-up questions
- When is a judge model the wrong tool, and a code check the right one?
- Why does a check that takes 500 ms belong online and not inline?
- How do you compare scores from two providers that use different scales?
- What should happen when a check itself fails: block or allow?