cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Core loop › Tool call › Exactly once

Agent Harness Tutorial · End to end · Zoom level 3

Doing a write exactly once

Networks fail at the worst moment: the claim was registered, but the reply got lost, so the browser sends the message again. Without care, Rohan now has two claims. The ledger makes every write happen exactly once.

Goes inA write with a key, e.g. run_fef1…:0:1
What happensClaim a row in the ledger before the call; record the result after
Comes outOne effect in the world, however many times it was asked
No rowclaim()in_flightexpires in 600 ssettle()answeredoutcome storedabandon(): no clear answer, so delete and let a retry ask againSame key againwhile in_flight→ 409, still runningSame key againafter answered→ the stored answer
The table is agent_state.requests in PostgreSQL. Reads never get a row.
WhereThe keyWhat a repeat gets
The message from the browserPH-1001 + the id the browser made for this messageThe first reply, or 409 if still running
A write tool callrun id : step : iterationThe stored result, "replayed"
The claims systemthe same key, carried in the MCP callThe first answer; no second claim, no second payout
An approvala hash of key + action + argumentsThe approval already running

Notice the key is carried all the way to the claims system, which keeps its own "answered" table. So even if our ledger were lost, the far end still refuses to do the same write twice.

Want every detail? The long version is in the Series: how idempotency works.

Code view: input, process, output, the call stack, the classes

Goes inA message's delivery id, and each write call's idempotency key
What happensEach name is claimed in agent_state.requests before the work runs, then settled with a definite answer or given back so a retry reuses the same name
Comes outOne effect per name; a repeat is handed the first answer
CALL STACK · START TO FINISHGOES IN → COMES OUT1Open the source: agent_harness/serve/__init__.py:252_run_turn(agent, inbound, conversation, span, gone=)agent_harness/serve/__init__.py:252in: decoded request with Idempotency-Key headerout: JSON reply; AlreadyAnswered replays 2002Open the source: claims_fnol/entrypoint/__init__.py:141Agent.handle(text, *, identity, conversation, delivery_id, …claims_fnol/entrypoint/__init__.py:141in: text and delivery idout: (TurnResult, Conversation)3Open the source: agent_harness/requests/__init__.py:156once(requests, name, *, scope=Scope.DELIVERY)agent_harness/requests/__init__.py:156in: name customer_id:delivery_idout: holds the claim around the turn4Open the source: agent_harness/requests/postgres.py:37PostgresRequests.claim(name, scope=Scope.DELIVERY)agent_harness/requests/postgres.py:37in: the delivery nameout: Claim, or AlreadyAnswered / StillRunning5Open the source: claims_fnol/entrypoint/__init__.py:178Agent._turn(text, identity, conversation, run_id, g…claims_fnol/entrypoint/__init__.py:178in: the turn's inputsout: result; claim.outcome = ending.as_answer6Open the source: agent_harness/requests/postgres.py:77PostgresRequests.settle(name, outcome)agent_harness/requests/postgres.py:77in: name and the turn's answerout: row set to answered, written once7Open the source: agent_harness/tools/__init__.py:166GatedTools._dispatch(name, arguments, identity, key, spec, spa…agent_harness/tools/__init__.py:166in: a non-read call and its keyout: result; AlreadyAnswered is replayed8Open the source: agent_harness/requests/__init__.py:156once(requests, key.value, *, scope=Scope.TOOL)agent_harness/requests/__init__.py:156in: name run_id:step:iterationout: Claim held for the call9Open the source: agent_harness/requests/postgres.py:37PostgresRequests.claim(name, scope=Scope.TOOL)agent_harness/requests/postgres.py:37in: the key valueout: Claim, or the first call's outcome10Open the source: agent_harness/tools/mcp.py:132MCPTransport.invoke(..., idempotency_key=key)agent_harness/tools/mcp.py:132in: call with the key in _metaout: the far end's result11Open the source: claims_system/store.py:235Store._write(operation, entity, holder, id, key, ac…claims_system/store.py:235in: the key, at the far endout: earlier answer for key, or new effect12Open the source: agent_harness/requests/postgres.py:77PostgresRequests.settle(name, outcome)agent_harness/requests/postgres.py:77in: a definite, non-transient resultout: row answered with the stored result13Open the source: agent_harness/requests/postgres.py:95PostgresRequests.abandon(name)agent_harness/requests/postgres.py:95in: an error or transient resultout: in-flight row deleted; retry uses name
Read top to bottom: each box is one function call, indented under the function that called it; the dark box is where it starts. Click a box to open its source file.
Dashed boxes are protocols (the ports) or enums; grey boxes are data models; white boxes are classes with behaviour. A dashed arrow means "implements". Click a class to open its source.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

agent_harness/requests/postgres.py:37Claim a ledger row: INSERT … ON CONFLICT
    async def claim(self, name: str, *, scope: Scope, ttl_s: int = CLAIM_TTL_S) -> Claim:
        """Take the name, or say which refusal this is.

        One statement. The `WHERE` on the conflict branch is the whole guard: it
        matches only a claim that expired, so a live holder and an answered name
        both leave zero rows and the second query below says which. Two callers
        racing here both attempt the same insert and exactly one changes a row.
        """
        async with self._pool.connection() as conn:
            taken = await (
                await conn.execute(
                    """
                    INSERT INTO agent_state.requests (name, scope, state, expires_at)
                    VALUES (%s, %s, 'in_flight', now() + make_interval(secs => %s))
                    ON CONFLICT (name) DO UPDATE
                        SET scope = EXCLUDED.scope,
                            state = 'in_flight',
                            outcome = NULL,
                            expires_at = EXCLUDED.expires_at,
                            recorded_at = now()
                      WHERE agent_state.requests.state = 'in_flight'
                        AND agent_state.requests.expires_at < now()
                    RETURNING name
                    """,
                    (name, scope.value, ttl_s),
                )
            ).fetchone()
            if taken is not None:
                return Claim(name=name, scope=scope)

            row = await (
                await conn.cursor(row_factory=dict_row).execute(
                    "SELECT state, outcome FROM agent_state.requests WHERE name = %s", (name,)
                )

Tech choices (what we chose, Azure and open-source options): L8 · Concurrency & flow