cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › Integrations

Eval Harness Tutorial · Architecture

Every tool we read from, call, or write to

Everything the framework touches, in four columns. We never replace any of these tools: we read from them, call them, or write into them. Each one sits behind an adapter, so adding a tool is one adapter plus one line of YAML, and the same rules run whichever tools a client has.

WE READ FROMwhere traces already areOTLP file (Collector)Open source · builtPhoenix (not yet run live)Open source · builtApp Insights (KQL)Azure · builtLangfuseOpen source · Tier 4LangSmith runsSaaS · Tier 8Promptfoo resultsOpen source · Tier 7Usage exports (cost)Any stack · Tier 5CloudWatch / X-RayAWS · laterCloud TraceGoogle Cloud · laterCALLS US INLINEhosts and hooksReference agent harnessOpen source · builtPydantic AI hooksOpen source · Tier 7LangGraph hooksOpen source · Tier 7LiteLLM guardrail (gateway)Open source · Tier 7Decorators (no framework)Open source · Tier 7CI step (GitHub, Azure DevOps)Any stack · Tier 7WE CALLchecks and judgesPresidio (PII)Open source · builtAzure AI EvaluationAzure · builtDeepEvalOpen source · builtRagasOpen source · builtJudge model (any)Any stack · builtOur five patternsOpen source · builtAzure Content SafetyAzure · Tier 6Prompt Guard 2 · safeguardOpen source · Tier 6Bedrock GuardrailsAWS · laterCost rules (6 of 36)Open source · builtWE WRITE TOyour existing toolsOTel gen_ai.evaluation.resultOpen source · builtLangfuse / Phoenix scoresOpen source · Tier 4App Insights eventsAzure · builtPrometheus + GrafanaOpen source · Tier 8LangSmith feedbackSaaS · Tier 8Alerts: Teams, ServiceNowAny stack · Tier 8Cube / Metric Views + MCPOpen source · Tier 9Dark: built. Blue: next. Dashed: planned (tier shown) or later.
Upstream on the left, downstream on the right. Each box says whose tool it is (open source or a cloud) and its state.
ToolHow we use itWhoseState
OTLP file (Collector)we read fromOpen sourceBuilt
Phoenix (not yet run live)we read fromOpen sourceBuilt
App Insights (KQL)we read fromAzureBuilt
Langfusewe read fromOpen sourceTier 4
LangSmith runswe read fromSaaSTier 8
Promptfoo resultswe read fromOpen sourceTier 7
Usage exports (cost)we read fromAny stackTier 5
CloudWatch / X-Raywe read fromAWSlater
Cloud Tracewe read fromGoogle Cloudlater
Presidio (PII)we callOpen sourceBuilt
Azure AI Evaluationwe callAzureBuilt
DeepEvalwe callOpen sourceBuilt
Ragaswe callOpen sourceBuilt
Judge model (any)we callAny stackBuilt
Our five patternswe callOpen sourceBuilt
Azure Content Safetywe callAzureTier 6
Prompt Guard 2 · safeguardwe callOpen sourceTier 6
Bedrock Guardrailswe callAWSlater
Cost rules (6 of 36)we callOpen sourceBuilt
Reference agent harnesscalls us inlineOpen sourceBuilt
Pydantic AI hookscalls us inlineOpen sourceTier 7
LangGraph hookscalls us inlineOpen sourceTier 7
LiteLLM guardrail (gateway)calls us inlineOpen sourceTier 7
Decorators (no framework)calls us inlineOpen sourceTier 7
CI step (GitHub, Azure DevOps)calls us inlineAny stackTier 7
OTel gen_ai.evaluation.resultwe write toOpen sourceBuilt
Langfuse / Phoenix scoreswe write toOpen sourceTier 4
App Insights eventswe write toAzureBuilt
Prometheus + Grafanawe write toOpen sourceTier 8
LangSmith feedbackwe write toSaaSTier 8
Alerts: Teams, ServiceNowwe write toAny stackTier 8
Cube / Metric Views + MCPwe write toOpen sourceTier 9
Per cloud, in one line

Open stack: OpenTelemetry, Phoenix or Langfuse, Presidio, an open judge model, Prometheus and Grafana. Azure: App Insights (KQL) in and out, Azure AI Evaluation and Content Safety, Azure OpenAI as the judge. AWS and Google Cloud: CloudWatch, Bedrock Guardrails and Cloud Trace come later, as adapters, when a client needs them.