Agent Harness Tutorial · Tech stack
L8 · Concurrency & flow
Who gets served first when load spikes. Family: Act · one turn, in order.
Why our choiceAzure gives the queue and the limits. Priority classes and honest shedding are ours.
| Option | Where from | Verdict | Why |
|---|---|---|---|
| APIM rate limit | ours | ✓ chosen | 30 calls a minute |
| One turn per conversation | ours | ✓ chosen | the ledger row, 600 s |
| Service Bus queue | ours | ✓ chosen | planned |
| Service Bus Premium | Azure | ◐ planned | priority, deadlines, dead-letter |
| APIM rate limit | Azure | ✓ used | per subscription |
| APIM llm-token-limit | Azure | ✗ dropped | not on the Consumption tier |
| Own semaphores | open source | · considered | |
| LiteLLM limits | open source | · considered | not picked |
How it works inside the harness: Exactly once.
Compared: What queues turns when load spikes?
| Option | From | Verdict | Ordering and priority | Dead-letter | Scales workers (KEDA) | Extra infrastructure | Status |
|---|---|---|---|---|---|---|---|
| Azure Service Bus Premium | Azure | chosen | sessions; priority via queues | built in | yes | a namespace | final (design) |
| PostgreSQL as queue (turns table) | ours | chosen | leases; one turn per chat | ours: failed status | yes, query-based | none | turn status, to build |
| Redis Streams | open source | not evaluated | per stream, consumer groups | ours: pending list | yes | Redis we must keep durable | Redis is for counters only |
| Azure Storage Queues | Azure | not evaluated | no ordering guarantee | no built-in dead-letter | yes | a storage account | too plain for deadlines |
| APIM rate limits | Azure | chosen | not a queue: 429 on overload | n/a | no | none: already there | built: 30 calls a minute |
DecisionThe design queues turns on Service Bus Premium for dead-lettering and worker scaling, while turn status, leases and the outbox live in PostgreSQL so a reply exists only if the turn committed. APIM rate limits sit in front as the last line of flow control.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
<!-- Request rate per subscription key (the agent's): 30 calls a minute. -->
<rate-limit calls="30" renewal-period="60" remaining-calls-header-name="x-ratelimit-remaining" />