Eval Harness Framework

One framework to check AI agents with any provider's evals, guardrails, cost and business rules.

Active · join meAI EngineeringForward-deployed ready

7 of 13 tasks done

1 · Purpose

The Eval Harness Framework checks AI agents for quality, security, cost and a business's own rules. It mixes checks from several providers in one run and gives one result format. It is for teams running agents who already use tools such as LangSmith, Langfuse or Application Insights and are tired of writing glue between them. It brings no database, UI or dashboards of its own: it reads from and writes back to the tools a team already has.

For: Teams running AI agents in production, AI platform teams, ops and quality leads.

2 · Business benefits

Write the glue onceOne adapter per provider replaces the custom code teams write each time they combine two eval tools.
Best check for each jobA team can take three checks from one provider and five from another, with one comparable result.
No new platform to buyResults go back into the team's existing tracing and dashboards, so nothing new to run or pay for.
Business rules, not just scoresOps teams write rules like "no cancel without an order lookup first" in YAML.
Cost and security in the same runCost rules and personal-data checks run alongside quality checks.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Add PAN and Aadhaar recognisers to the Presidio adapter
  • Write contract test rows for one adapter
  • Review the claims example YAML as a newcomer

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Trace sourceReads conversations from wheretraces already land.PlanThe YAML that says which checksrun, where, and with whatEvaluator adaptersTranslate in, call oneprovider's check, translateJudgeThe model used by checks thatneed one.Decide stepTurns results into one action:allow, redact, block, escalateResult sinkWrites results back to theteam's own tracing tool.conversationschecks to runjudge callsactions and sc
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Evaluator adapters → Decide step: results

6 · Tech stack, per platform

ComponentOpen stackAzureAWSGCP
Trace sourceOpenAI-format messages ✓; Phoenix, Langfuse (planned)Application Insights (planned)——
PlanYAML ✓———
Evaluator adaptersDeepEval, Ragas, Presidio ✓Azure AI Evaluation groundedness ✓; Content Safety (planned)Bedrock evaluations and guardrails (planned)planned, no work started
JudgeGroq-hosted open model ✓———
Result sinkOpenTelemetry evaluation events (planned)Application Insights custom events (planned)——

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Tier 0: spike 2 of 2 done

TaskStatusSizeSkill
Run four providers through one request shapedoneMPython
Record every provider's real answer for offline testsdoneSTesting

Tier 1: core package 3 of 4 done

TaskStatusSizeSkill
Check the competitor landscape with sourcesdoneMResearch
Extract the package and use it from both reference agentsdoneMPython packaging
Contract test table per portdoneSTesting
A stability label on every adaptertodoSPython

Tier 2: sessions and domain patterns 2 of 3 done

TaskStatusSizeSkill
Session shape and five domain patternsdoneMPython
Worked example for the claims agentdoneSYAML
Run the five patterns on real claims and support conversationstodoMAI evaluation

Tiers 3 to 6: version 1 slice 0 of 4 done

TaskStatusSizeSkill
Read traces from Phoenix and Application InsightstodoMObservability
Write results back as OpenTelemetry evaluation eventstodoMOpenTelemetry
Bring in the six built LLM cost rulestodoMCost engineering
Add Indian ID formats (PAN, Aadhaar) to the personal-data checktodoSSecurity

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Adapter developer23-5 hours/weekAdds and maintains one provider's adapter and its contract tests.
Rule author12-3 hours/weekWrites domain patterns and cost rules in YAML.
Pilot team contact11-2 hours/weekRuns version 1 on a real team's traces and reports what they found.

9 · What you'll learn

Level-up questions

10 · Links