AI engineering · cloud binding

The agent harness on AWS

The same harness, the same specs and the same tests as on Azure. Only the binding changes: each box gets an AWS service, an open-source tool, or our own code.

The one-minute version. An agent that talks to dealers, customers and staff needs much more than a model: sign-in, guardrails, memory, tools, approvals, logging, cost limits, tests. We call that the harness, and we split it into 22 concerns.

The concerns never name a product. For each cloud we write a binding: which service does each job. This page is the AWS binding, with a tyre manufacturer as the example. On average, by our judgement, about a third of each concern comes from AWS (36%), nearly half is shared code we write once (45%), and about a fifth is what each agent team writes (19%).

Already seen the Azure version? Compare the harness on Azure and the Azure tech stack, concern by concern. Why we work from specs: spec-driven development.

22 concernsin five families: think, act, control, see, change
6 agentsfor dealers, plants, quality and fleets, on one shared platform, each with its own autonomy level
12 horizontal assetsbuilt once on AWS, reused by every agent
India firstMumbai, with Hyderabad for disaster recovery; a Frankfurt copy of the same stack only when EU personal data arrives

The big picture

One request, top to bottom, on AWS

Read it as one request: a dealer or an engineer asks, through a channel, past the front door, to one vertical agent, which uses the shared assets to reach the company's systems. The left rail is the life of every agent; the right rail is governance.

End users · who asksDealersCognito / portalFleet customersCognitoConsumersCognitoPlant & qualityIdentity CenterSales & serviceIdentity CenterChannels · where they askDealer portal chatChatwoot or ownWhatsAppEnd User MessagingEmailAmazon SESContact centreAmazon ConnectStaff chatTeams or SlackFront door · checks every requestEdgeCloudFront + WAFHidden orders?Bedrock GuardrailsPersonal data?Comprehend PIIIn scope?our router rulesFair shareAPI Gateway limitsOrchestration · on ECS FargateRouterour codeA2A hand-offA2A over HTTPSAgent registryGit + ownerVertical agents · tyre manufacturer · one ECS service eachDealer assistantstock, price, orders: answersWarranty claim triagedrafts for warranty deskPlant maintenancesuggests, from manualsQuality deviationdrafts root cause noteB2B order assistantacts after OKFleet tyre advisoranswers, from telematics↓ every agent reuses the same assets ↓Horizontal assets · built once on AWS, reused by every agentModel gatewayAgentCore GatewayKnowledgepgvector on RDSMemory & stateRDS PostgreSQLTool / MCP hubMCP on FargateChannelsConnect, SES, EUMIdentityCognito, IAM, KMSGuardrailsBedrock + our rulesHuman deskDBOS + ConnectObservabilityOpenTelemetry → ADOTCost controlprofiles, BudgetsEval libraryBedrock, AgentTwinReleaseAppConfig, ECRSystems of record · via MCP and PrivateLinkSAP S/4HANAorders, stock, pricingDealer managementDMSMESplant executionHistorian + IoTIoT SiteWiseQuality · LIMSWarranty systemCRMSupplier portalLIFECYCLEIdea & risk tieris AI the right fix?BuildGitHub ActionsEvaluateBedrock + AgentTwinRed-teamPyRIT, promptfooApprovebusiness ownerReleaseECS blue/greenMonitorCloudWatchLearnmisses → golden setGOVERNANCEOrganizationsSCPs, Control TowerCloudTrailwho did whatSecurity Hub+ GuardDutyIdentity Centeraccess reviewsResidencyIndia first; EU laterDPDP + GDPRconsent, erasureOwner & risk tierin the registryAI councilsigns off high risk
The dark cell is the first agent to build (the dealer assistant). The vertical-agents band changes per business; every other band is shared. Compare with the same picture on Azure: the boxes do not move, only the names change.
BandWhat it does on AWS
End usersDealers, fleet customers and consumers sign in through Cognito or the existing dealer portal; plant, quality, sales and service staff through IAM Identity Center.
ChannelsWeb and app chat, WhatsApp and SMS through AWS End User Messaging, email through SES, and the contact centre through Amazon Connect.
Front doorCloudFront and WAF at the edge; Bedrock Guardrails for hidden instructions; Comprehend to find personal data; our router rules; API Gateway limits.
OrchestrationOur router picks one agent; agents hand work to each other over A2A; a registry in Git names the owner and version of each.
Vertical agentsOne ECS service per business job. Each has an autonomy level: answers, suggests, drafts for a person, or acts after checks.
Horizontal assetsTwelve shared parts every agent reuses (next sections).
Systems of recordSAP S/4HANA, the dealer management system, MES, the plant historian and IoT SiteWise, quality and LIMS, warranty, CRM and the supplier portal, reached only through MCP tools with the user's own rights.

End to end

The design, from account to release

This is a reference architecture, so every choice below is made. The tyre maker is only the example. Read the drawing first, then follow one request through it, then the nine design tiers (H0 to H8).

PEOPLE AND CHANNELSDealer portal chatour pageWhatsApp, SMSEnd User MessagingEmailAmazon SESContact centreConnect (optional)StaffIdentity CenterCloudFront + WAFthe one way inCognitoportal OIDCagents-prod account · Mumbai · private VPC, 2 zonesLoad balancerthe only public partECS on FargateRouter + 6 agentsour loop, Pydantic AIADOT collectorOpenTelemetry sidecarToken issueron-behalf-of, KMS keyMCP tool serversSAP, DMS, MES, warrantyChatwoot desk runs here tooRDS for PostgreSQLstate · DBOS · pgvectorSQS FIFOone group per conversationPrivate endpoints (PrivateLink)AgentCore Gateway→ Bedrock (Claude)Bedrock Guardrails+ Comprehend PIISecrets + KMSAppConfigkill switchX-Ray + CloudWatchvia OTLPNetwork Firewall: outbound only to an allow-listeverything built by TerraformOUTSIDE THE ACCOUNTTransit Gatewaynetwork accountPlants and SAPDirect Connect / VPNData platform accountDatabricks SQL, PrivateLinkHyderabadRDS replica for DRLog-archive accountCloudTrail, model logsShared servicesGitHub OIDC, ECR
The production account in Mumbai. The dark box is our code: the router, the loop and the agents. Everything telemetry-related flows through OpenTelemetry into the ADOT sidecar, then on to X-Ray and CloudWatch. Dashed arrow: the replica in Hyderabad.

One request, step by step

#WhereWhat happens
1DealerA dealer in Pune asks in the portal chat: "Can I get 40 of 205/55 R16 by Friday, and at what price?" The portal's sign-in gives a token (Cognito, federated).
2EdgeCloudFront and WAF filter the request; the load balancer passes it to the router on ECS.
3RouterChecks the token, runs the input guardrail (ApplyGuardrail), opens the turn in PostgreSQL (one turn per conversation) and picks the dealer assistant.
4ContextThe assembler builds the turn: recent conversation from PostgreSQL, the dealer's own price list passages from pgvector.
5ModelThe loop calls the model through AgentCore Gateway to Claude on Bedrock in Mumbai; the gateway checks the allow-list and the token budget.
6ToolsThe model asks for check_stock and get_price. Gating checks the arguments and the dealer scope; our issuer mints a 5-minute token on the dealer's behalf; the SAP MCP server calls SAP over Direct Connect.
7Way outOur rules check the reply: the price comes from this dealer's list, the date from SAP's confirmed stock. The grounding check confirms nothing was invented.
8ReplyThe dealer gets stock, price and date. "Place the order" would start an approval: a DBOS wait and a card to the sales desk.
9TraceEvery step was an OpenTelemetry span sent to the ADOT sidecar, then on to X-Ray and CloudWatch; the model call is logged to S3; the meter wrote the cost of the turn.

H0 · Foundation

The plot of land: accounts, region, private network, keys, and the code that builds it all.

DecisionWe useNot usedWhy
AccountsAWS Organizations + Control Tower: agents-dev, agents-test, agents-prod, plus shared security, log-archive, network and shared-services accountsone account for everythingAgents face the internet; a problem in one account cannot reach another. The data platform has its own accounts in the same organisation.
RegionMumbai (ap-south-1) primary, Hyderabad (ap-south-2) for disaster recovery; a policy (SCP) blocks every other region except us-east-1 for global servicesactive-active across regionsIndian personal data stays in India. A Frankfurt copy is the same Terraform with another region, built only when EU personal data arrives.
NetworkOne VPC per account across 2 zones, private subnets only; the load balancer is the only public part; Transit Gateway in the network account; outbound only through Network Firewall with a domain allow-listopen NAT egressOne way in, one way out, both inspected.
Private pathsVPC endpoints for Bedrock, AgentCore, STS, KMS, Secrets Manager, CloudWatch Logs, X-Ray, ECR, SQS, AppConfig; S3 gateway endpointpublic AWS endpointsNo call to the model or to a secret crosses the internet.
KeysKMS customer keys per kind of data: agent state, logs and transcripts, knowledgeone key for allErasing or rotating one kind of data leaves the others alone.
BlueprintTerraform for everything, shared modules with the data platformAWS CDKOne tool for AWS and Databricks (it has a Terraform provider).

H1 · Model door

Every model call goes through one governed door, so models can be swapped and spend can be capped.

DecisionWe useNot usedWhy
GatewayBedrock AgentCore Gateway in front of Bedrock; LiteLLM on Fargate as the fallback behind the same portagents calling Bedrock directlyThe counterpart of APIM's AI gateway: identity, limits, fallback and logging in one place. The agent never holds a model key.
ModelsClaude on Bedrock: Sonnet for reasoning, Haiku for routing and short jobs; the model per job lives in AppConfig. The reference implementation's dev setup uses a free open-weight model (Groq) behind the same gatewayIntelligent Prompt RoutingPicking the model is a decision we test offline with the golden set, not one the platform makes per prompt.
ResidencyCalls carrying Indian personal data run in Mumbai; no Asia-Pacific cross-region profile for themAPAC cross-region inference for personal dataThat profile can route to Tokyo, Singapore or Sydney.
SpendOne application inference profile per agent (tagged), a token budget per turn, prompt caching on the fixed part of the context—Cost per agent comes from AWS; cost per business case comes from our meter (H7).

H2 · Runtime and loop

Where our code runs, and the loop that plans, acts and checks.

DecisionWe useNot usedWhy
ComputeECS on Fargate: one service for the router, one per agent, one per MCP tool serverAgentCore Runtime, EKS, LambdaNo cluster to run, the closest match to Container Apps. AgentCore Runtime gets a pilot later; Lambda's cold starts and limits fit a 60-second turn badly.
LoopOur hand-written loop (about 400 lines): 12 steps, 60 seconds and a cost cap per turn; Pydantic AI only as the model layerBedrock Agents, Strands, LangGraphThe loop is where every check has an exact position; it stays ours on every cloud.
Model layerPydantic AI under our loop: typed calls, structured output and one interface for Bedrock and GroqClaude Agent SDK, raw boto3 ConverseThe Agent SDK is Claude-only and runs its own loop out of sight. Calling boto3 directly would mean rebuilding typing, retries and output checks. Pydantic AI helps without taking over the loop.
FrontProduction: CloudFront + WAF → Application Load Balancer. Dev: API Gateway HTTP API, with the channel's webhook answered before the turn runs—API Gateway caps a request at 30 seconds, below the 60-second turn.
FlowSQS FIFO with one message group per conversation; one turn per conversation at a time; priority classes—Order is kept per conversation; a plant breakdown goes before a marketing question.
RegistryA catalogue file in Git: agent, version, owner, autonomy levelAWS Agent RegistryNot offered in Mumbai; Git is enough.

H3 · State, memory, knowledge

What the agent keeps, for how long, for whom, and where it finds sourced answers.

DecisionWe useNot usedWhy
DatabaseAmazon RDS for PostgreSQL, Multi-AZ: one instance, schemas agent_state, dbos and knowledge; one database role per appAurora Serverless v2, DynamoDBPostgreSQL, the same engine as on Azure. Serverless v2 never pauses while DBOS holds a connection, so plain RDS is cheaper and simpler.
MemoryConversation and case tables; retention per data class; erase per person (DPDP Act)AgentCore MemoryMemory must follow our retention and erasure rules.
Knowledgepgvector in the same database, Bedrock embeddings; the entitlement filter is a SQL WHERE clauseBedrock Knowledge Bases, OpenSearchA dealer sees only their own price list and orders; one store, one backup, one access model.
When to outgrow itStay on pgvector (HNSW index) up to roughly 10 million passages or when searches start slowing turns; then move knowledge to OpenSearch Serverless or S3 Vectors behind the same retrieval port—State and memory are ordinary rows and stay in PostgreSQL; only knowledge search would move.
RecoveryAutomated backups + AWS Backup; a read replica in Hyderabad, promoted in a disaster (back within 4 hours, at most 15 minutes lost)Aurora Global DatabaseEnough for the stated targets at a fraction of the cost.

H4 · Tools to systems of record

How agents reach SAP, the dealer system, the plants and the data platform, with the user's own rights.

DecisionWe useNot usedWhy
Tool serversOne MCP server per system on Fargate: SAP S/4HANA, dealer management, MES and historian, warranty, CRM. Read tools first; write tools only behind an approvalagents calling systems directlyThe protocol is open, so the servers move clouds unchanged.
GatingBefore every call: argument check (JSON Schema), scope check (this dealer's records only), idempotency key, size cap—Protects orders, prices and warranties; ours on every cloud.
On behalf ofOur token issuer mints a 5-minute token for the far system on behalf of the user, signed with a KMS key; tool servers check it against our published keys (JWKS)AgentCore IdentityThe same pattern as the Azure FNOL agent; Cognito has no on-behalf-of exchange.
SAPSAP's OData APIs over Direct Connect or VPN through Transit Gatewayreading SAP's databaseSAP's own rules and locks stay in force.
Data platformOne MCP tool over a Databricks SQL warehouse (Metric Views) through PrivateLink, with the user's Unity Catalog rightsreading raw S3 filesNumbers come from the governed semantic layer.

H5 · Identity, privacy, guardrails

Who is asking, what data the turn may use, and the checks on the way in and out.

DecisionWe useNot usedWhy
PeopleCognito user pool for dealers, fleets and consumers, federating the dealer portal over OIDC; staff through IAM Identity Center—Dealers keep one login; the harness only needs a verified token.
ServicesIAM task roles; secrets in Secrets Manager encrypted with KMSkeys in code or environmentNothing to leak.
Way inBedrock Guardrails through the ApplyGuardrail API (prompt attacks, denied topics) on user input and on tool resultsNeMo GuardrailsOne detector port; works with any model.
Way outOur 30 positioned rules (no price or warranty outside policy, no order changed without OK, a dealer's own records only) + contextual grounding check—Every rule that protects money or a customer is ours.
Personal dataComprehend PII plus our recognisers for Aadhaar, PAN, GSTIN and vehicle numbers, redacting before telemetry; Macie on the buckets—Telemetry and logs never carry raw personal data.

H6 · Channels and the human desk

How people reach the agent, and how a person takes over or approves.

DecisionWe useNot usedWhy
Web chatOur chat page inside the dealer portal—Simple, and the portal already has the sign-in.
Other channelsWhatsApp and SMS through AWS End User Messaging; email through SES; one conversation id across channels—Dealers in India live on WhatsApp.
DeskChatwoot on Fargate as the human desk; Amazon Connect adapter when a contact centre existsbuilding our own deskA desk with the full history, without building one.
ApprovalsDBOS durable waits in the same PostgreSQL; an approval card to the sales or warranty desk; autonomy level per agentStep Functions task tokensSurvives restarts, same code as on Azure, no extra service.

H7 · Observability, replay, cost

What we can see, prove and pay for. Everything goes through OpenTelemetry.

DecisionWe useNot usedWhy
Telemetry standardOpenTelemetry SDK in our code, one helper, GenAI semantic conventions; personal data redacted before exportAWS SDK calls for telemetryThe code never changes between clouds; only the collector's config does.
CollectorAWS Distro for OpenTelemetry (ADOT) as a sidecar in every ECS task: traces to X-Ray, metrics and logs to CloudWatch—AWS's supported build of the OpenTelemetry collector.
Dashboards and alertsCloudWatch Application Signals for SLOs; alarms as code (Terraform) to SNS email and chat—Turn latency at the 95th percentile, error rate, guardrail blocks, cost per hour.
EvidenceBedrock model invocation logging to S3 in the log-archive account (1 year); CloudTrail for who changed what—Answers an auditor's question months later.
ReplayRun record + cassettes + config fingerprint in S3—"Why did the agent tell this dealer that price?" needs a replay; no cloud offers one.
CostInference profile tags + Budgets + CUR 2.0; our meter per turn and per case with a hard budget stop—Cost per warranty claim triaged, not just per agent.

H8 · Evaluation and release

How a change is proven safe and rolled out, and rolled back.

DecisionWe useNot usedWhy
CIGitHub Actions with OIDC into AWS (no stored keys); four gates: AgentTwin world, golden sets, red team (PyRIT, promptfoo), cost and latencyCodePipelineThe same gates as on Azure.
ImagesAmazon ECR, scanned on pushpublic registries in productionPrivate images, close to Fargate.
ReleaseOne release bundle (prompt + model + policy + config fingerprint); ECS blue/green through CodeDeploy with alarms that roll back—Prompt, model and policy change together and roll back together.
Runtime configAWS AppConfig: kill switch per agent, limits, model per job, read before every turn—A bad behaviour is switched off in seconds, without a deploy.
Online evaluationEach night a sample of live turns is graded; Bedrock evaluation is one judge plug-in; misses go to the golden set—The golden set keeps learning from production.

Every concern, bound to AWS

The 22 concerns, one table

For each concern: what AWS offers and whether we use it, the open-source options, and what we code once for every agent, and why.

use in the design   planned next phase   considered not picked, or newer service to check   not used and why

ConcernOn AWSOpen sourceOurs, built once, and why
Think · how the agent decides
L1 · Context assembly
What the model sees each turn, built in one place, within a budget.
  • not usedBedrock Prompt Management stores prompts, does not assemble a turn
  • useJinja templates fixed replies only
Our assembler: cache-friendly order, fencing, trimming from the middle, transcript check; prompts in Git
No product on any cloud assembles a turn the way a regulated agent needs, so we code it once.
L2 · Model invocation
Every model call goes through one door.
  • useAmazon Bedrock, Converse API Claude, Llama, Mistral, Nova behind one API
  • useCross-region inference profiles keep inside the India geography where offered; check per model
  • not usedBedrock Intelligent Prompt Routing picks the model per prompt; not deterministic
  • useBedrock AgentCore Gateway, inference targets any provider incl. OpenAI-compatible ones like Groq; translates OpenAI, Anthropic and Bedrock dialects; model fallback with circuit breaker; request and token-per-minute limits per user, model or target; Bedrock Guardrails in gateway policy (checked 3 Nov 2026)
  • useBedrock application inference profiles one per agent, tagged, so cost and quota are per agent
  • consideredLiteLLM proxy on ECS Fargate fallback gateway if AgentCore Gateway lacks a rule we need, or for non-AWS models
  • usePydantic AI model layer only
Our gateway policy: allow-list per job, token budget, one retry budget, token metric per agent
AgentCore Gateway is AWS's AI gateway, the same job APIM's AI gateway does on Azure: one governed door with identity, rate and token limits, fallback and logging. Our policy (allow-list per job, budgets) sits on top as config. Check region availability and token-limit features before go-live. The agent never holds a provider key.
L4 · Control loop
Plan, act, check, with step and no-progress limits.
  • not usedBedrock Agents the platform would own the loop
  • consideredBedrock AgentCore Runtime could host our own loop; pilot later
  • not usedAgentCore managed harness AWS's own harness (CreateHarness, GA 2026): AWS runs the loop, memory and evaluation; our loop is the kept delta
  • not usedStrands Agents AWS's open SDK; fine for simple agents
  • not usedLangGraph workflow-shaped agents only
  • not usedClaude Agent SDK Claude-only, loop hidden
Hand-written loop, about 400 lines; budgets of 12 steps, 60 s and a cost cap per turn
Agents that touch orders, warranties and money need every step testable and checks at exact positions. Our own loop runs on any model and any cloud.
L5 · State & memory
What is kept, for how long, and for whom.
  • useAmazon RDS for PostgreSQL, Multi-AZ HA, backups, IAM login; one instance holds state, DBOS waits and pgvector
  • consideredAurora PostgreSQL when one RDS instance is not enough; Serverless v2 never pauses while DBOS holds a connection
  • consideredDynamoDB not picked
  • consideredElastiCache for Valkey caches only
  • consideredAgentCore Memory check current status
  • usepsycopg + pool driver
5 agent tables + DBOS state in the same database; retention and erase per person
Plain RDS PostgreSQL, the same engine as on Azure. One managed store that also holds the approval engine and, later, pgvector: no extra infrastructure.
18 · Knowledge & retrieval
Finds sourced passages, filtered by what this user may see.
  • plannedpgvector on the same RDS PostgreSQL chosen
  • consideredBedrock Knowledge Bases managed RAG; less control of filters
  • consideredOpenSearch Serverless if the corpus outgrows pgvector
  • consideredS3 Vectors cheap cold vectors; check current status
  • plannedpgvector chosen
  • consideredQdrant on ECS alternative
Entitlement filter as SQL: a plant engineer sees machine manuals, a dealer sees only their own orders and price lists
One store, one backup, one access model; the entitlement check is just a SQL filter.
Act · how it reaches people and systems
CH · Channels & human desk
How dealers, customers and staff reach the agent, and how a person takes over with the full history.
  • plannedAWS End User Messaging SMS and WhatsApp
  • plannedAmazon SES email
  • plannedAmazon Connect contact centre, chat and hand-off to a person
  • consideredAPI Gateway WebSocket push replies to the window
  • plannedChatwoot web chat + desk if there is no contact centre
  • consideredA Next.js window alone six more things to build
Our chat page and desk API; one conversation id across channels
If the company already runs a contact centre, hand off into it (Connect). Otherwise Chatwoot gives the desk without building one.
L3 · Tool layer
Actions through typed tools, checked before they run.
  • useMCP servers on ECS Fargate or Lambda one server per system of record
  • plannedBedrock AgentCore Gateway the same gateway fronts our MCP tool servers: one door for tools and models
  • useMCP Python SDK client + servers
  • consideredFastMCP
Gating before every call, idempotency keys, size caps, argument checks with jsonschema
The tool protocol is open (MCP), so the servers move clouds unchanged. The gating that protects orders, prices and warranties is ours.
L6 · I/O contracts
Typed inputs and outputs at every edge.
  • plannedAPI Gateway request validation reject bad shapes at the edge
  • usePydantic
  • useJSON Schema
  • plannedSAP OData / BAPI shapes typed at the ERP adapter
Frozen Pydantic models everywhere; MCP structured output
Contracts are code, not cloud. SAP and dealer-system shapes are typed once at their adapters.
L8 · Concurrency & flow
Who gets served first when load spikes.
  • plannedAmazon SQS FIFO one message group per conversation keeps order
  • useAPI Gateway usage plans rate limits per client
  • useBedrock quotas tokens per minute, the real ceiling
  • consideredOwn semaphores
  • consideredLiteLLM limits only if LiteLLM is the gateway
One turn per conversation (ledger row), priority classes (plant breakdowns before marketing), honest shedding
AWS gives the queue and the limits. Which request waits, and what we tell the person, is ours.
17 · Agent-to-agent
Agents hand work to agents through typed, bounded, traced hand-offs.
  • plannedA2A over HTTPS behind API Gateway
  • consideredAgentCore A2A support check current status
  • plannedA2A SDK
  • consideredBackstage registry
Registry in Git: who, which version, owner, autonomy level
A2A is an open protocol, so hand-offs work across clouds. The registry stays a file we own.
PS · Performance & scalability
Fast enough at the 95th percentile; more servers when load rises; the model quota is the real limit.
  • useECS Fargate service auto scaling
  • useBedrock prompt caching cheaper, faster repeated context
  • plannedDistributed Load Testing on AWS AWS solution
  • consideredBedrock provisioned throughput only at steady high volume
  • consideredk6 / Locust
Per-turn time budget (60 s), cache-friendly context order
Servers are stateless and scale out; the model quota is what to plan and load-test.
Control · what keeps it safe
L16 · Identity & authz
Who is asking, and whose authority the agent acts with.
  • plannedAmazon Cognito dealer, fleet and consumer sign-in, or federate the dealer portal
  • useIAM Identity Center staff sign-in
  • useIAM roles for tasks no keys in code
  • useSecrets Manager + KMS
  • consideredAmazon Verified Permissions Cedar policies for fine-grained rules
  • usePyJWT local token check
  • consideredKeycloak
Principal → dealer, customer or staff identity; on-behalf-of: the agent acts with the user's rights, never its own
Checking a token is local and cloud-neutral. Ports make leaving Cognito a configuration change.
PV · Privacy & data protection
Personal data of dealers, fleet drivers and consumers used only for this case, kept in its country, masked, and forgotten on request.
  • plannedComprehend PII names, phone, ids in text
  • plannedAmazon Macie finds personal data in S3
  • useMumbai (Hyderabad for DR) India data stays in India
  • plannedFrankfurt a second copy of the stack, only when EU personal data arrives
  • useKMS, PrivateLink
  • consideredPresidio Aadhaar, PAN, GSTIN and vehicle numbers need custom recognisers anyway
redact() in telemetry (Aadhaar, PAN, phone, vehicle number), no_pii_echo rule, forget() + retention per the DPDP Act and GDPR
Residency and encryption come from AWS. Which data a turn may use, masking and erasure are domain rules we code once.
L7 · Policy enforcement
Guardrails on the way in and on the way out.
  • useBedrock Guardrails prompt attacks, harmful content, denied topics, PII
  • useApplyGuardrail API the same checks on any model and on tool results
  • plannedContextual grounding check answers must stay on the sources
  • consideredAutomated Reasoning checks check current status
  • plannedLlama Prompt Guard 2 small local classifier
  • not usedNeMo Guardrails wants to own the loop
  • not usedLlama Guard an LLM, too slow
Our positioned rules (30 checks): no price or warranty promised outside policy, no order changed without OK, a dealer's own records only
We own every position and every rule; every check that protects money or a customer is ours. AWS supplies detectors behind one port.
L10 · Failure handling
Typed failures, fallbacks, undo paths, honest messages.
  • useCross-region inference spreads load when a region is busy
  • useALB / ECS health checks
  • consideredtenacity
  • useOwn resilience wrapper one retry budget per turn
Typed failures, ledger abandon-and-retry, an honest message instead of a guess
Retries without a budget hide outages. One retry budget per turn is ours on any cloud.
L14 · Human-in-the-loop
Approvals and hand-offs the agent cannot grant itself.
  • consideredStep Functions task tokens a managed wait, now callable as an MCP tool through the Gateway; but a second state store, and AWS-only
  • plannedAmazon Connect hand-off to a person with history
  • useDBOS durable waits in our own PostgreSQL
  • consideredTemporal
Approval and escalation tables, desk API, approval cards to staff
A warranty or order approval must survive restarts. DBOS keeps it in the database we already run, so it moves clouds unchanged.
AV · Availability & disaster recovery
A lost server or zone goes unnoticed; a lost region is rebuilt from code and backups.
  • useMulti-AZ ECS + RDS Multi-AZ
  • plannedRDS cross-region read replica Mumbai → Hyderabad, promoted in a disaster
  • plannedRoute 53 health checks
  • plannedAWS Backup
  • useTerraform or AWS CDK infrastructure as code
Stateless servers + durable state; a rebuild drill
Two Indian regions make a DR pair without data leaving the country. Target: back within 4 hours, at most 15 minutes of data lost.
See · what we can see and prove
L11 · Observability
Traces, metrics and alerts for every step.
  • useAWS Distro for OpenTelemetry (ADOT) collector a sidecar in every ECS task; receives OTLP from our code, sends traces to X-Ray, metrics and logs to CloudWatch
  • useCloudWatch + Application Signals dashboards, SLOs, alarms
  • useAWS X-Ray traces, read through the OTLP endpoint
  • useBedrock model invocation logging to S3, for evidence
  • consideredCloudWatch generative AI observability check current status
  • plannedManaged Prometheus + Grafana
  • useOpenTelemetry SDK + GenAI semantic conventions the only telemetry API our code calls
  • consideredLangfuse takes OTLP too, if a team wants an LLM trace viewer
One telemetry helper on the OpenTelemetry SDK: spans per step with GenAI attributes, personal data redacted before export, alert rules as code
OpenTelemetry is the one standard from code to collector: the code never imports an AWS SDK for telemetry. On Azure the same spans go to Application Insights; on AWS the ADOT collector sends them to X-Ray and CloudWatch. Moving clouds changes the collector's config file, not the code.
L9 · Determinism & replay
A past decision can be re-run and explained.
  • considered(nothing on AWS) invocation logs help, but do not replay
  • consideredVCR.py
  • useOwn recorder
Run record + cassettes + config fingerprint
"Why did the agent say that to this dealer?" needs a replay. No cloud offers it, so it is ours.
L13 · Cost accounting
Spend measured per unit of business work.
  • useBedrock application inference profiles one tagged profile per agent = cost per agent
  • useAWS Budgets
  • useCost Explorer
  • plannedCUR 2.0 data exports
  • consideredLangfuse cost
  • useOwn meter
Meter per turn and per case (cost per warranty claim triaged), hard budget stop
AWS tells you cost per agent; only we know cost per warranty claim or per dealer order.
Change · how it changes safely
L12 · Eval harness
Golden sets and graders that gate every release.
  • plannedBedrock model evaluation LLM-as-judge and human review
  • plannedBedrock RAG evaluation
  • plannedAgentCore Evaluations one more evaluator provider inside our eval framework (includes DeepEval evaluators; online evaluation of live traces)
  • useAgentTwin, pytest, pydantic-evals
  • plannedpromptfoo, PyRIT red team
Golden sets owned by quality, warranty and dealer-service leads, hard graders, 4 CI gates, model selection run
The pretend dealer network and plant (AgentTwin) and the hard graders are the same on any cloud; AWS judges are one plug-in.
L15 · Release & config
Prompt, model and policy versioned and rolled back together.
  • useAWS AppConfig kill switch, limits, model per job
  • useECS blue/green deployments
  • useAmazon ECR
  • consideredCodePipeline or GitHub Actions + OIDC
  • useGitHub Actions + OIDC
  • consideredOpenFeature
One release bundle: prompt + model + policy + config fingerprint
AppConfig is a close match for App Configuration; the bundle and rollback rule are ours.

Services marked "check current status" are newer AWS offerings; confirm their availability in Mumbai and Hyderabad (and Frankfurt, if the EU copy is built) before relying on them.

Who builds what

What AWS gives, what we build, what each agent writes

For each concern, how much does AWS give us, how much do we code once for every agent, and how much does each agent team still write? AWS gives the plumbing. The parts that make an agent safe for dealers and customers stay in our code.

AWS giveswe code once (horizontal asset)each agent team writesL16 · Identity & authz702010AV · Availability & disaster recovery652510L2 · Model invocation553510PS · Performance & scalability553510L11 · Observability553510CH · Channels & human desk503515L8 · Concurrency & flow503515L15 · Release & config50351518 · Knowledge & retrieval453025PV · Privacy & data protection454015L5 · State & memory404515L7 · Policy enforcement404020L13 · Cost accounting355510L3 · Tool layer253540L10 · Failure handling255520L12 · Eval harness25453017 · Agent-to-agent206020L14 · Human-in-the-loop205525L6 · I/O contracts105040L1 · Context assembly6035L4 · Control loop8020L9 · Determinism & replay8515
Our judgement, not a measurement. AWS gives the plumbing (gateway, identity, storage, logging); the parts that make an agent trustworthy stay in our code.

The rule that keeps this healthy: when two agents write the same code, it moves into the shared middle (blue). When AWS catches up, it moves to the left (grey).

The five big choices

Decided, with the options we dropped

DecisionOptionsWe choseVerdictWhy
Which model?Claude on Bedrock · Llama or Nova on Bedrock · a model outside AWS via the gatewayClaude on Bedrock as the default for every job, behind our gateway. The golden set picks the model per job offline; cheaper models win where they pass.Pursue: Claude on BedrockData stays in the AWS account; switching model is one line in AppConfig.
Which gateway (the one door to models)?Bedrock AgentCore Gateway · LiteLLM proxy on ECS Fargate · API Gateway + Lambda (build it ourselves)AgentCore Gateway, with our policy (allow-list, budgets) as its rules; LiteLLM on Fargate kept as the fallback behind the same port.Pursue: AgentCore GatewayManaged by AWS, the closest match to APIM's AI gateway, and it also fronts MCP tools. LiteLLM only if a rule we need is missing or a non-AWS model is required. Check Mumbai availability first.
Where does our code run?ECS Fargate · Bedrock AgentCore Runtime · EKSECS Fargate: the closest match to Azure Container Apps, no cluster to run.Pursue: ECS FargateAgentCore Runtime is worth a pilot later; EKS only if the company already runs Kubernetes.
How do approvals wait?DBOS on RDS PostgreSQL · Step Functions task tokensDBOS: the same code as on Azure, state in the PostgreSQL we already run.Pursue: DBOSStep Functions works, but it is one more service and AWS-only.
How do dealers and customers sign in?Amazon Cognito · federate the existing dealer portal over OIDCCognito user pool, federating the dealer portal over OIDC.Pursue: Cognito + portal federationDealers keep one login; the harness only needs a verified token.

Moving clouds

Azure to AWS in one table

JobOn AzureOn AWS
Front doorAzure Front Door + WAFCloudFront + AWS WAF
AI gatewayAPI Management (AI gateway)Bedrock AgentCore Gateway (LiteLLM on Fargate as fallback)
ModelsFoundry ModelsAmazon Bedrock (Converse API)
Where code runsAzure Container AppsAmazon ECS on Fargate
Sign-inEntra ID, External IDIAM Identity Center (staff), Cognito or portal OIDC (dealers, fleets, consumers)
Secrets and keysKey Vault, managed identitySecrets Manager + KMS, IAM task roles
GuardrailsAI Content Safety, Prompt ShieldsBedrock Guardrails (ApplyGuardrail)
Finding personal dataAI Language PIIComprehend PII, Macie for S3
DatabaseAzure Database for PostgreSQLAmazon RDS for PostgreSQL
QueueService BusSQS FIFO
Traces and metricsOpenTelemetry → Application InsightsOpenTelemetry → ADOT collector → X-Ray + CloudWatch
Runtime config, kill switchApp ConfigurationAWS AppConfig
CostCost Management, APIM token metricBudgets, Cost Explorer, Bedrock inference-profile tags
EvaluationFoundry evaluation, AI Red Teaming AgentBedrock evaluations, PyRIT, promptfoo
Infrastructure as codeBicep + azdTerraform or AWS CDK
Contact centreCommunication Services, DynamicsAmazon Connect, End User Messaging, SES

What does not change: specs, tests and agent code.

  • The 22 concerns and the specs (AI Harness Catalog, AI Assurance Catalog, agent specs)
  • The agent code: router, loop, context assembler, tools, approval flows
  • The tests: AgentTwin worlds, golden sets, hard graders, the four CI gates
  • The rules: every guardrail position and every check that protects money or a customer
  • The ports: each AWS service sits behind the same interface the Azure one did

That is the point of binding through ports: moving from Azure to AWS is a new binding file and a set of adapters, not a rebuild.

Finally

What we are using

The whole binding on one page: for each concern, only what we picked. AWS services and open-source tools marked planned are chosen but not yet switched on; everything else in the tables above was considered and left out.

ConcernAWS serviceOpen sourceOur own code
Think
L1 · Context assembly—Jinja templatesOur assembler: cache-friendly order, fencing, trimming from the middle, transcript check; prompts in Git
L2 · Model invocationAmazon Bedrock, Converse API
Cross-region inference profiles
Bedrock AgentCore Gateway, inference targets
Bedrock application inference profiles
Pydantic AIOur gateway policy: allow-list per job, token budget, one retry budget, token metric per agent
L4 · Control loop——Hand-written loop, about 400 lines; budgets of 12 steps, 60 s and a cost cap per turn
L5 · State & memoryAmazon RDS for PostgreSQL, Multi-AZpsycopg + pool5 agent tables + DBOS state in the same database; retention and erase per person
18 · Knowledge & retrievalpgvector on the same RDS PostgreSQL (planned)pgvector (planned)Entitlement filter as SQL: a plant engineer sees machine manuals, a dealer sees only their own orders and price lists
Act
CH · Channels & human deskAWS End User Messaging (planned)
Amazon SES (planned)
Amazon Connect (planned)
Chatwoot (planned)Our chat page and desk API; one conversation id across channels
L3 · Tool layerMCP servers on ECS Fargate or Lambda
Bedrock AgentCore Gateway (planned)
MCP Python SDKGating before every call, idempotency keys, size caps, argument checks with jsonschema
L6 · I/O contractsAPI Gateway request validation (planned)Pydantic
JSON Schema
SAP OData / BAPI shapes (planned)
Frozen Pydantic models everywhere; MCP structured output
L8 · Concurrency & flowAmazon SQS FIFO (planned)
API Gateway usage plans
Bedrock quotas
—One turn per conversation (ledger row), priority classes (plant breakdowns before marketing), honest shedding
17 · Agent-to-agentA2A over HTTPS behind API Gateway (planned)A2A SDK (planned)Registry in Git: who, which version, owner, autonomy level
PS · Performance & scalabilityECS Fargate service auto scaling
Bedrock prompt caching
Distributed Load Testing on AWS (planned)
—Per-turn time budget (60 s), cache-friendly context order
Control
L16 · Identity & authzAmazon Cognito (planned)
IAM Identity Center
IAM roles for tasks
Secrets Manager + KMS
PyJWTPrincipal → dealer, customer or staff identity; on-behalf-of: the agent acts with the user's rights, never its own
PV · Privacy & data protectionComprehend PII (planned)
Amazon Macie (planned)
Mumbai (Hyderabad for DR)
Frankfurt (planned)
KMS, PrivateLink
—redact() in telemetry (Aadhaar, PAN, phone, vehicle number), no_pii_echo rule, forget() + retention per the DPDP Act and GDPR
L7 · Policy enforcementBedrock Guardrails
ApplyGuardrail API
Contextual grounding check (planned)
Llama Prompt Guard 2 (planned)Our positioned rules (30 checks): no price or warranty promised outside policy, no order changed without OK, a dealer's own records only
L10 · Failure handlingCross-region inference
ALB / ECS health checks
Own resilience wrapperTyped failures, ledger abandon-and-retry, an honest message instead of a guess
L14 · Human-in-the-loopAmazon Connect (planned)DBOSApproval and escalation tables, desk API, approval cards to staff
AV · Availability & disaster recoveryMulti-AZ ECS + RDS Multi-AZ
RDS cross-region read replica (planned)
Route 53 health checks (planned)
AWS Backup (planned)
Terraform or AWS CDKStateless servers + durable state; a rebuild drill
See
L11 · ObservabilityAWS Distro for OpenTelemetry (ADOT) collector
CloudWatch + Application Signals
AWS X-Ray
Bedrock model invocation logging
Managed Prometheus + Grafana (planned)
OpenTelemetry SDK + GenAI semantic conventionsOne telemetry helper on the OpenTelemetry SDK: spans per step with GenAI attributes, personal data redacted before export, alert rules as code
L9 · Determinism & replay—Own recorderRun record + cassettes + config fingerprint
L13 · Cost accountingBedrock application inference profiles
AWS Budgets
Cost Explorer
CUR 2.0 data exports (planned)
Own meterMeter per turn and per case (cost per warranty claim triaged), hard budget stop
Change
L12 · Eval harnessBedrock model evaluation (planned)
Bedrock RAG evaluation (planned)
AgentCore Evaluations (planned)
AgentTwin, pytest, pydantic-evals
promptfoo, PyRIT (planned)
Golden sets owned by quality, warranty and dealer-service leads, hard graders, 4 CI gates, model selection run
L15 · Release & configAWS AppConfig
ECS blue/green deployments
Amazon ECR
GitHub Actions + OIDCOne release bundle: prompt + model + policy + config fingerprint

How the code fits

Library dependencies

Agents hold only their domain code. Everything else comes from two libraries we own and a handful of open-source ones. Moving to AWS adds one install option, [aws], to each library: new adapters behind the same ports, nothing else.

AGENTS · domain code onlyclaims_fnolmotor claims agent (built)dealer assistanttyre maker (next)config/<env>.yamllocal · azure · aws (new)SPECS AND CATALOGS · data, not codeGuardrail catalogrule ids, placesAOAS · AHC · AACwhat must be truestacks/azure.yaml · stacks/aws.yamlagent-harness · the core library (unchanged)loopcontexttools + gatingpolicyapprovalscost metertelemetrywatchreplayports (adapters/): model · state · identity · secrets · configtelemetry · waits · tools · records · cost ledgerevalharness · eval frameworkports: Evaluator · TraceSourceJudge · ResultSinkdomain patterns · cost rulesfive positions, input to releaseagenttwin · pretend worldstests and gates onlyOPEN SOURCE UNDERNEATHPydantic AImodel layerDBOSdurable waitsMCP SDKtoolsOpenTelemetrySDK + OTLPpsycopgPostgreSQLPydanticcontractsStarletteHTTP edgeCLOUD EXTRAS · pip install agent-harness[azure] or [aws], evalharness[azure] or [aws][azure] · builtmodel → APIM AI gatewaystate → Azure PostgreSQL · sign-in → Entra IDsecrets → Key Vault · config → App Configurationtelemetry → Azure Monitor · guardrails → Content Safetyevals → Azure AI evaluation, App Insights traces[aws] · new, step R1model → AgentCore Gateway → Bedrockstate → RDS PostgreSQL · sign-in → Cognito + our issuersecrets → Secrets Manager · config → AppConfigtelemetry → OTLP → ADOT · guardrails → Bedrock Guardrailsevals → AgentCore Evaluations, CloudWatch traces
Solid arrows: depends on. Dashed from the agent to agenttwin: tests only, never at run time. Dashed into [azure]: the extra we built. The yellow box is the only new code for AWS.

AgentCore, part by part

Which AgentCore parts we use

AgentCore is AWS's set of building blocks for agents. We take the parts that are plumbing and keep the parts that make our agents ours.

AgentCore partVerdictWhyWhat of ours sits there
GatewayUseThe one door to models and MCP tools: any provider, fallback, request and token limits per user, guardrails at the door.Our gateway policy (allow-list per job, budgets) rides on it
PolicyUseCedar rules on who may call which tool; Bedrock Guardrails at the gateway for detector-type checks.Rules from our guardrail catalog that are detector-shaped
ObservabilityUseShows our OpenTelemetry traces in CloudWatch; nothing to install in our code.Our telemetry helper + ADOT
EvaluationsUse, as a plug-inOne more evaluator provider inside our eval framework, including online evaluation of live traces.evalharness decides what runs where
IdentityNot nowToken vault and OAuth to third-party tools. Add it when a SaaS tool needs a user's consent.Our on-behalf-of token issuer for SAP and the dealer system
MemoryNoOur memory must follow our retention and erasure rules and sit next to the approvals.Memory tables in RDS PostgreSQL
RuntimeNot nowManaged hosting for agent containers; worth a pilot later.ECS on Fargate
Managed harnessNoAWS runs the loop, memory and evaluation for you.Our loop, the part we keep on every cloud
Code Interpreter, BrowserNoNo tyre-company job needs them yet.—
Agent RegistryNoNot offered in Mumbai.Catalogue file in Git

Reuse, don't rebuild

What we already built, and how it lands on AWS

AssetWhat it isOn AWS
Agent harness library (agent_harness)Loop, context assembler, tool gating, DBOS approvals, telemetry helper, ports and adapters, the four offline gates with AgentTwinUnchanged. Add the AWS adapters (model, state, identity, secrets, config, telemetry); each passes the contract tests the Azure ones pass. Offline gates run in GitHub Actions and need no cloud.
Guardrail catalogEvery check, with its id, what it catches and where it may runThe catalog stays the source of truth. Each rule is bound to a place: our code inside the loop (most rules), Bedrock Guardrails at the gateway for detector-type checks (prompt attack, harmful content, PII), the eval framework for online and release checks. Same rule id everywhere, so one report covers all places.
Eval framework (evalharness)Evaluator, TraceSource, Judge and ResultSink ports; domain patterns; cost rules; five positionsAdd AWS adapters: a trace source that reads OpenTelemetry spans from CloudWatch, Bedrock Guardrails and AgentCore Evaluations as evaluator providers, Bedrock as a judge model, and a result sink that writes scores back as OpenTelemetry evaluation events to CloudWatch. Nightly online run as a scheduled ECS task.
Trace viewer (the LangSmith job)We never built our own; local dev uses self-hosted LangfuseProduction: CloudWatch generative AI observability, fed by the same ADOT pipeline, with our scores written next to the traces. Langfuse on ECS is an optional second destination for teams that want prompt playground and annotation queues; adding it is a collector config change.

AWS's gateway gives the door: any model, fallback, rate and token limits per user, guardrails at the door. It does not give spend in money per turn or per business case, a retry budget per turn, rules at exact positions inside the loop, or replay. Those stay ours.

From design to running code

The reference implementation

The same path the motor-claims FNOL agent took on Azure: write the binding, write the adapters, add a configuration overlay, build the infrastructure, run it, then add the next agent. The agent's code and tests do not change.

StepWhatHowState
R0Binding written downclean-ai-engineering/stacks/aws.yaml: every port named with its AWS adapterdone on paper (19 Oct)
R1AWS adaptersIn the agent_harness library: agentcore-gateway, rds-postgresql, cognito + our token issuer, secrets-manager, appconfig, otel → ADOT; a contract test per port, the same tests the Azure adapters passdone (3 Nov), tested offline; live recordings wait for an AWS account
R2The FNOL agent on AWS, unchangedAdd config/aws.yaml to the motor-claims FNOL agent: a new overlay, no change to the agent's code. This proves that only the binding changesdone (3 Nov): one profile file, two overlays, keys published at the edge; no agent code changed
R3Infrastructure as codeTerraform for H0 to H8 at dev size (one account, budget alarm), checked offline firstplanned
R4Running on AWSDeploy to AWS dev; first claim end to end, traced in X-Ray through ADOTplanned
R5Tyre dealer assistantThe second agent on the same stack: SAP and dealer-system MCP servers against an AgentTwin dealer worldplanned

Read next

Go deeper

Agent harness tutorial (every concern explained, on the Azure example) · Spec-driven development · Data platform architecture · Guardrail catalog · AgentTwin.

To walk through this with your team, write to basant@cleandataengineering.ai.