Eval Harness Tutorial · From traces
What the standard can't say yet
The OpenTelemetry standard for AI agents (the GenAI conventions) has names for model calls, tool calls, messages, tokens and conversations. It has no name for a decision such as handing over to a person, and none for how a conversation ended.
So a session rebuilt from standard traces has no decision steps,
and decision_vs_should can't run on it until a team says
which of its own attributes means “handed over”. The reference agent records this as
termination; another team will use another name.
| What | In the standard? | In our reader |
|---|---|---|
| Conversation id | Yes: gen_ai.conversation.id | Built |
| Messages in and out | Yes | Built |
| Tool calls, arguments, results, errors | Yes | Built |
| A hand-over or approval decision | No | Not built yet map a team's own attribute in YAML |
| How the conversation ended | No | Not built yet |
OpenInference, the naming Phoenix's own instrumentors use, has span kinds DECISION and GUARDRAIL. Our Phoenix reader translates OpenInference model and tool spans today; reading its decision spans as hand-overs is the next step, and evidence for the OpenTelemetry proposal.
Two things: let a team map its own attribute to a decision in YAML, and add these gaps to our proposal to the OpenTelemetry group (Tier 11). The standard itself is still marked “in development”, so this is a good moment to propose it.
- Should we propose a hand-over field to OpenTelemetry now, or wait until a second team needs it?
Decided: Wait for a second team. Meanwhile a team maps its own attribute to a decision in YAML.