Agent Harness Tutorial · End to end · Zoom level 2
The trace
Every turn produces one trace: a tree of timed pieces called spans. The tree mirrors the boxes on the end-to-end map, so you can find any box of any turn after the fact.
No customer words. On Azure, Rohan's words and the replies are not captured. Routes, timings, tokens, costs and check verdicts are. A redaction step also removes things like Aadhaar, PAN, card and phone numbers before anything leaves the process.
One helper. Every module records spans through one small function. If recording fails, the agent carries on: watching must never change what the agent does.
Ids on spans, not on counters. Counters (turns, refusals per rule, tool calls) only carry labels with a few known values, so dashboards stay cheap.
The trace has the same shape for a real turn, a test scenario and a simulated run. There is no separate "eval-only" code path.
Code view: input, process, output, the call stack, the classes
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
agent_harness/telemetry/__init__.py:334The one helper every module records spans with@contextmanager
def span(name: str, **attributes: Any) -> Iterator[Span]:
"""A span with attributes set at open, and errors recorded rather than
swallowed. Exceptions propagate — observability never changes control flow."""
with _tracer().start_as_current_span(name) as current:
working_for = _WORKING_FOR.get()
if working_for is not None and SESSION_ID not in attributes:
current.set_attribute(SESSION_ID, working_for)
for key, value in attributes.items():
if value is not None:
current.set_attribute(key, value)
try:
yield current
except Exception as exc:
current.set_status(StatusCode.ERROR, str(exc))
current.record_exception(exc)
raiseagent_harness/telemetry/redaction.py:112redact(): personal data out before anything leavesdef redact(text: str, *, limit: int = 4000) -> str:
"""The single redaction point. Everything captured onto a span comes here.
Deliberately crude: deterministic regexes, no model call, no network. A
detector that can be wrong slowly is worse than one that is obviously
approximate — this is a floor, not a compliance control.
"""
for pattern, replacement in (*_LIBRARY, *_registered):
text = pattern.sub(replacement, text)
if len(text) > limit:
return text[:limit] + f"…[truncated {len(text) - limit} chars]"
return textconfig/azure.yaml:29The telemetry port on Azure telemetry:
adapter: azure-monitor-otel
connection_string: {key_vault: appinsights-connection-string}Tech choices (what we chose, Azure and open-source options): PV · Privacy & data protection · L11 · Observability