cleandataengineering.aiAgent Harness, drawn simply
You are here: Tech stack › L8 · Concurrency & flow

Agent Harness Tutorial · Tech stack

L8 · Concurrency & flow

Who gets served first when load spikes. Family: Act · one turn, in order.

AZURE OPTIONSService Bus Premium◐ planned: priority, deadlines, dead-letterAPIM rate limit✓ used: per subscriptionAPIM llm-token-limit✗ dropped: not on the Consumption tierOUR CHOICEAPIM rate limit30 calls a minuteOne turn per conversationthe ledger row, 600 sService Bus queueplannedOPEN-SOURCE OPTIONSOwn semaphores· consideredLiteLLM limits· considered: not pickedWHO DOES THE WORKAzure gives 50% · we code once 35% · each agent team still does 15%
Dark boxes are what we run. Solid boxes are used or planned; dashed boxes were considered or dropped (crossed). The bar: grey is what Azure gives, dark is what we code once, hatched is what each agent team writes.
Why our choiceAzure gives the queue and the limits. Priority classes and honest shedding are ours.
OptionWhere fromVerdictWhy
APIM rate limitours✓ chosen30 calls a minute
One turn per conversationours✓ chosenthe ledger row, 600 s
Service Bus queueours✓ chosenplanned
Service Bus PremiumAzure◐ plannedpriority, deadlines, dead-letter
APIM rate limitAzure✓ usedper subscription
APIM llm-token-limitAzure✗ droppednot on the Consumption tier
Own semaphoresopen source· considered
LiteLLM limitsopen source· considerednot picked

How it works inside the harness: Exactly once.

Compared: What queues turns when load spikes?

Azure Service Bus Pre…AzurechosenPostgreSQL as queue (…ourschosenRedis Streamsopen sourcenot evaluatedAzure Storage QueuesAzurenot evaluatedAPIM rate limitsAzurechosen
Dark: what we chose. Crossed and dashed: rejected or not for us. Added here (not a Series table).
OptionFromVerdictOrdering and priorityDead-letterScales workers (KEDA)Extra infrastructureStatus
Azure Service Bus PremiumAzurechosensessions; priority via queuesbuilt inyesa namespacefinal (design)
PostgreSQL as queue (turns table)ourschosenleases; one turn per chatours: failed statusyes, query-basednoneturn status, to build
Redis Streamsopen sourcenot evaluatedper stream, consumer groupsours: pending listyesRedis we must keep durableRedis is for counters only
Azure Storage QueuesAzurenot evaluatedno ordering guaranteeno built-in dead-letteryesa storage accounttoo plain for deadlines
APIM rate limitsAzurechosennot a queue: 429 on overloadn/anonone: already therebuilt: 30 calls a minute
DecisionThe design queues turns on Service Bus Premium for dead-lettering and worker scaling, while turn status, leases and the outbox live in PostgreSQL so a reply exists only if the turn committed. APIM rate limits sit in front as the last line of flow control.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

    <!-- Request rate per subscription key (the agent's): 30 calls a minute. -->
    <rate-limit calls="30" renewal-period="60" remaining-calls-header-name="x-ratelimit-remaining" />