cleandataengineering.aiEval Harness, drawn simply
You are here: From traces › Gaps in the standard

Eval Harness Tutorial · From traces

What the standard can't say yet

The OpenTelemetry standard for AI agents (the GenAI conventions) has names for model calls, tool calls, messages, tokens and conversations. It has no name for a decision such as handing over to a person, and none for how a conversation ended.

So a session rebuilt from standard traces has no decision steps, and decision_vs_should can't run on it until a team says which of its own attributes means “handed over”. The reference agent records this as termination; another team will use another name.

WhatIn the standard?In our reader
Conversation idYes: gen_ai.conversation.idBuilt
Messages in and outYesBuilt
Tool calls, arguments, results, errorsYesBuilt
A hand-over or approval decisionNoNot built yet map a team's own attribute in YAML
How the conversation endedNoNot built yet
One standard already has it

OpenInference, the naming Phoenix's own instrumentors use, has span kinds DECISION and GUARDRAIL. Our Phoenix reader translates OpenInference model and tool spans today; reading its decision spans as hand-overs is the next step, and evidence for the OpenTelemetry proposal.

What we'll do about it

Two things: let a team map its own attribute to a decision in YAML, and add these gaps to our proposal to the OpenTelemetry group (Tier 11). The standard itself is still marked “in development”, so this is a good moment to propose it.

Decided · Gaps in the standard
  1. Should we propose a hand-over field to OpenTelemetry now, or wait until a second team needs it?
    Decided: Wait for a second team. Meanwhile a team maps its own attribute to a decision in YAML.