cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Core loop › Model call

Agent Harness Tutorial · End to end · Zoom level 2

Model call

The loop calls the model through one tiny socket with one method: send a request, get a response. Everything that can go wrong on the way (rate limits, timeouts, a blocked prompt) is turned into a clear, named result before it reaches the loop.

Goes inThe assembled messages and the list of tools
What happensLayers count and retry; APIM checks; Groq runs the model
Comes outA typed response: words, tool requests, tokens used, and why the model stopped
The loopcomplete(request)Open: Layers around the callLayers around the callcount tokens, retry,trip a breakerzoom in ›Open: APIM gatewayAPIM gatewayAzure checkszoom in ›Groqopenai/gpt-oss-120bback: text and/or tool requests, token counts, why it stopped
Two boxes you can open: the layers around the call, and the APIM gateway.

What goes out. The model name, temperature 0 (so it is as repeatable as possible), at most 4,096 tokens back, the messages from context assembly, and the 8 tools Rohan may use, each with a name, a description and the shape of its inputs.

What comes back. Some text, or some tool requests, or both; the tokens used; and the stop reason. If the stop reason is "length" (the model ran out of room), the answer is thrown away, never sent on half-finished.

What it costs. Every response is priced at once: $0.15 per million tokens in, $0.75 per million out. If the turn reaches $0.50 it stops. A model with no price fails at start-up; it is never priced as free.

Code view: input, process, output, the call stack, the classes

Goes ina ModelRequest: messages, tools, max_tokens
What happensThe request passes through four wrappers to the OpenAI SDK, goes over HTTP to APIM, and the reply is parsed back into a typed ModelResponse on the way out.
Comes outa ModelResponse, or a typed model failure
CALL STACK · START TO FINISHGOES IN → COMES OUT1Open the source: agent_harness/loop/__init__.py:244_Run._ask(step_span) # self.llm.complete(request)agent_harness/loop/__init__.py:244in: ModelRequestout: ModelResponse or a failed Ended2Open the source: claims_fnol/entrypoint/plain.py:49PlainText.complete(request)claims_fnol/entrypoint/plain.py:49in: ModelRequestout: response with plain hyphens and spaces3Open the source: claims_fnol_app/usage.py:43Counted.complete(request)claims_fnol_app/usage.py:43in: ModelRequestout: the same response; calls and tokens counted4Open the source: agent_harness/resilience/__init__.py:289ResilientLLM.complete(request)agent_harness/resilience/__init__.py:289in: ModelRequestout: response, or the last typed failure5Open the source: agent_harness/resilience/__init__.py:116with_retry(partial(self._attempt, ...), retry_on…agent_harness/resilience/__init__.py:116in: one attempt functionout: the first successful response6Open the source: agent_harness/resilience/__init__.py:308ResilientLLM._attempt(request, tries)agent_harness/resilience/__init__.py:308in: request and attempt counterout: response, or _Retry wrapping the failure7Open the source: agent_harness/resilience/__init__.py:235Throttle.wait(self._sleep)agent_harness/resilience/__init__.py:235in: a shared retry-after deadlineout: waits if the provider asked us to slow down8Open the source: agent_harness/llm/pydantic_ai.py:232PydanticAIClient.complete(request)agent_harness/llm/pydantic_ai.py:232in: ModelRequestout: ModelResponse with gateway signals9Open the source: agent_harness/llm/gateway.py:66gateway.heard()agent_harness/llm/gateway.py:66in: nothingout: a dict that APIM response headers fill10Open the source: agent_harness/llm/pydantic_ai.py:78to_messages(request.messages) # and …agent_harness/llm/pydantic_ai.py:78in: harness Messages and tool definitionsout: Pydantic AI messages and ToolDefinitions11model_request(self._model, ..., instr…library: pydantic_ai.directin: OpenAIChatModel, messages, settingsout: pa.ModelResponse12AsyncOpenAI chat.completions # b…library: openaiin: HTTPS POST to APIM base_url, key in key_headerout: the chat completion JSON from APIM13Open the source: agent_harness/llm/gateway.py:76_on_response(response) # htt…agent_harness/llm/gateway.py:76in: the APIM HTTP responseout: x- header signals stored in heard14Open the source: agent_harness/llm/pydantic_ai.py:133from_response(raw)agent_harness/llm/pydantic_ai.py:133in: pa.ModelResponseout: ModelResponse, or ModelMalformed15Open the source: agent_harness/llm/pydantic_ai.py:283_record(span, standard, response, too…agent_harness/llm/pydantic_ai.py:283in: response and time takenout: gen_ai metrics and span attributes
Read top to bottom: each box is one function call, indented under the function that called it; the dark box is where it starts. Click a box to open its source file.
Dashed boxes are protocols (the ports) or enums; grey boxes are data models; white boxes are classes with behaviour. A dashed arrow means "implements". Click a class to open its source.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

agent_harness/contracts/protocols.py:30The model socket: one method
@runtime_checkable
class LLMClient(Protocol):
    """AHC-0022 — provider interaction is substitutable without changing the
    system. One choke point, every call passes through it."""

    async def complete(self, request: ModelRequest) -> ModelResponse: ...
class ModelRequest(BaseModel):
    model_config = ConfigDict(frozen=True)

    messages: tuple[Message, ...]
    tools: tuple[dict[str, object], ...] = ()
    max_tokens: int = 4096
    temperature: float | None = None
    """None takes the client's configured temperature, which is zero by default:
    determinism is the floor, variance is opted into. A number is this call's
    own. It was `0.0` meaning *unset*, so asking for zero under a non-zero
    configuration was read as not asking (F-064)."""
class ModelResponse(BaseModel):
    model_config = ConfigDict(frozen=True)

    text: str = ""
    tool_calls: tuple[ToolCall, ...] = ()
    usage: Usage = Field(default_factory=Usage)
    model: str = ""
    stop_reason: str = ""
    gateway: dict[str, str] = Field(default_factory=dict)
    """What a gateway in front of the provider said about this call, by
    lower-cased header name (`x-content-safety: pass; prompt=pass; …`). Empty
    when there is no gateway or it said nothing (`llm.gateway.SIGNALS`)."""

    @property
    def wants_tools(self) -> bool:
        return len(self.tool_calls) > 0
config/azure.yaml:32Which plug fills the socket on Azure
  model:
    adapter: apim-ai-gateway
    base_url: {env: APIM_GROQ_BASE_URL}
    api_key: {key_vault: apim-subscription-key}

Tech choices (what we chose, Azure and open-source options): L2 · Model invocation

Look inside