cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › The trace

Agent Harness Tutorial · End to end · Zoom level 2

The trace

Every turn produces one trace: a tree of timed pieces called spans. The tree mirrors the boxes on the end-to-end map, so you can find any box of any turn after the fact.

http.chatthe requestagent.turnwho, which conversation, resultagent.routekind + reasonagent.runcost, why it endedagent.stepcontext size, trimmed?gen_ai.chatmodel, tokens, safety verdictagent.policywhich check, blocked by?agent.tooltool, key, outcomeApplication Insights38 rows for the"my claim" turnkept 30 daysOpenTelemetry
Indentation shows which span is inside which. The model span follows the OpenTelemetry standard for AI calls.

No customer words. On Azure, Rohan's words and the replies are not captured. Routes, timings, tokens, costs and check verdicts are. A redaction step also removes things like Aadhaar, PAN, card and phone numbers before anything leaves the process.

One helper. Every module records spans through one small function. If recording fails, the agent carries on: watching must never change what the agent does.

Ids on spans, not on counters. Counters (turns, refusals per rule, tool calls) only carry labels with a few known values, so dashboards stay cheap.

The trace has the same shape for a real turn, a test scenario and a simulated run. There is no separate "eval-only" code path.

Code view: input, process, output, the call stack, the classes

Goes inA turn's work, and the Application Insights connection string
What happensAt startup the Azure Monitor distro installs the providers around the harness's own processors; during the turn each layer opens a named span and any captured words pass through redaction first
Comes outA trace of nested spans in Application Insights and in memory
CALL STACK · START TO FINISHGOES IN → COMES OUT1Open the source: agent_harness/adapters/telemetry.py:59_azure_monitor(wiring)agent_harness/adapters/telemetry.py:59in: the overlay's connection_stringout: the in-memory exporter2Open the source: agent_harness/adapters/telemetry.py:31_configured(wiring, **extra)agent_harness/adapters/telemetry.py:31in: capture settings and backendout: result of tel.configure3Open the source: agent_harness/telemetry/__init__.py:138configure(*, capture_payloads, ..., backend)agent_harness/telemetry/__init__.py:138in: service name and capture choiceout: InMemorySpanExporter; provider installed4Open the source: agent_harness/telemetry/__init__.py:203_install(resource, processors, ..., backend)agent_harness/telemetry/__init__.py:203in: resource, exporter and RunNumbersout: the TracerProvider the backend built5Open the source: agent_harness/telemetry/azure.py:43install(connection, resource, processors, reader)agent_harness/telemetry/azure.py:43in: connection string and processorsout: providers from configure_azure_monitor6Open the source: agent_harness/serve/__init__.py:203chat(request)agent_harness/serve/__init__.py:203in: POST /chatout: span http.chat7Open the source: claims_fnol/entrypoint/__init__.py:178Agent._turn(text, identity, conversation, run_id, gone)claims_fnol/entrypoint/__init__.py:178in: the turnout: tel.turn_scope, then span agent.turn8Open the source: claims_fnol/router/__init__.py:249route(text, *, rules)claims_fnol/router/__init__.py:249in: the messageout: span agent.route9Open the source: agent_harness/loop/__init__.py:95run(goal, *, identity, llm, tools, ...)agent_harness/loop/__init__.py:95in: the agentic goalout: span agent.run (+ agent.tools.list)10Open the source: agent_harness/loop/__init__.py:203_Run.step(step)agent_harness/loop/__init__.py:203in: one stepout: spans agent.step, gen_ai.chat, agent.policy11Open the source: agent_harness/tools/__init__.py:124GatedTools.call(name, arguments, identity, idemp…agent_harness/tools/__init__.py:124in: one tool callout: span agent.tool with args and result12Open the source: agent_harness/telemetry/__init__.py:314span(name, **attributes)agent_harness/telemetry/__init__.py:314in: a span name and attributesout: the open Span; errors recorded13Open the source: agent_harness/telemetry/__init__.py:329set_payload(current, key, text, *, limit)agent_harness/telemetry/__init__.py:329in: words to captureout: attribute set only if this turn captures14Open the source: agent_harness/telemetry/redaction.py:112redact(text, *, limit)agent_harness/telemetry/redaction.py:112in: raw textout: text with ids, cards, tokens masked
Read top to bottom: each box is one function call, indented under the function that called it; the dark box is where it starts. Click a box to open its source file.
Dashed boxes are protocols (the ports) or enums; grey boxes are data models; white boxes are classes with behaviour. A dashed arrow means "implements". Click a class to open its source.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

agent_harness/telemetry/__init__.py:334The one helper every module records spans with
@contextmanager
def span(name: str, **attributes: Any) -> Iterator[Span]:
    """A span with attributes set at open, and errors recorded rather than
    swallowed. Exceptions propagate — observability never changes control flow."""
    with _tracer().start_as_current_span(name) as current:
        working_for = _WORKING_FOR.get()
        if working_for is not None and SESSION_ID not in attributes:
            current.set_attribute(SESSION_ID, working_for)
        for key, value in attributes.items():
            if value is not None:
                current.set_attribute(key, value)
        try:
            yield current
        except Exception as exc:
            current.set_status(StatusCode.ERROR, str(exc))
            current.record_exception(exc)
            raise
agent_harness/telemetry/redaction.py:112redact(): personal data out before anything leaves
def redact(text: str, *, limit: int = 4000) -> str:
    """The single redaction point. Everything captured onto a span comes here.

    Deliberately crude: deterministic regexes, no model call, no network. A
    detector that can be wrong slowly is worse than one that is obviously
    approximate — this is a floor, not a compliance control.
    """
    for pattern, replacement in (*_LIBRARY, *_registered):
        text = pattern.sub(replacement, text)
    if len(text) > limit:
        return text[:limit] + f"…[truncated {len(text) - limit} chars]"
    return text
config/azure.yaml:29The telemetry port on Azure
  telemetry:
    adapter: azure-monitor-otel
    connection_string: {key_vault: appinsights-connection-string}

Tech choices (what we chose, Azure and open-source options): PV · Privacy & data protection · L11 · Observability

Look inside