Eval Harness Tutorial · Architecture
Every tool we read from, call, or write to
Everything the framework touches, in four columns. We never replace any of these tools: we read from them, call them, or write into them. Each one sits behind an adapter, so adding a tool is one adapter plus one line of YAML, and the same rules run whichever tools a client has.
| Tool | How we use it | Whose | State |
|---|---|---|---|
| OTLP file (Collector) | we read from | Open source | Built |
| Phoenix (not yet run live) | we read from | Open source | Built |
| App Insights (KQL) | we read from | Azure | Built |
| Langfuse | we read from | Open source | Tier 4 |
| LangSmith runs | we read from | SaaS | Tier 8 |
| Promptfoo results | we read from | Open source | Tier 7 |
| Usage exports (cost) | we read from | Any stack | Tier 5 |
| CloudWatch / X-Ray | we read from | AWS | later |
| Cloud Trace | we read from | Google Cloud | later |
| Presidio (PII) | we call | Open source | Built |
| Azure AI Evaluation | we call | Azure | Built |
| DeepEval | we call | Open source | Built |
| Ragas | we call | Open source | Built |
| Judge model (any) | we call | Any stack | Built |
| Our five patterns | we call | Open source | Built |
| Azure Content Safety | we call | Azure | Tier 6 |
| Prompt Guard 2 · safeguard | we call | Open source | Tier 6 |
| Bedrock Guardrails | we call | AWS | later |
| Cost rules (6 of 36) | we call | Open source | Built |
| Reference agent harness | calls us inline | Open source | Built |
| Pydantic AI hooks | calls us inline | Open source | Tier 7 |
| LangGraph hooks | calls us inline | Open source | Tier 7 |
| LiteLLM guardrail (gateway) | calls us inline | Open source | Tier 7 |
| Decorators (no framework) | calls us inline | Open source | Tier 7 |
| CI step (GitHub, Azure DevOps) | calls us inline | Any stack | Tier 7 |
| OTel gen_ai.evaluation.result | we write to | Open source | Built |
| Langfuse / Phoenix scores | we write to | Open source | Tier 4 |
| App Insights events | we write to | Azure | Built |
| Prometheus + Grafana | we write to | Open source | Tier 8 |
| LangSmith feedback | we write to | SaaS | Tier 8 |
| Alerts: Teams, ServiceNow | we write to | Any stack | Tier 8 |
| Cube / Metric Views + MCP | we write to | Open source | Tier 9 |
Per cloud, in one line
Open stack: OpenTelemetry, Phoenix or Langfuse, Presidio, an open judge model, Prometheus and Grafana. Azure: App Insights (KQL) in and out, Azure AI Evaluation and Content Safety, Azure OpenAI as the judge. AWS and Google Cloud: CloudWatch, Bedrock Guardrails and Cloud Trace come later, as adapters, when a client needs them.