AI engineering · cloud binding
The agent harness on AWS
The same harness, the same specs and the same tests as on Azure. Only the binding changes: each box gets an AWS service, an open-source tool, or our own code.
The one-minute version. An agent that talks to dealers, customers and staff needs much more than a model: sign-in, guardrails, memory, tools, approvals, logging, cost limits, tests. We call that the harness, and we split it into 22 concerns.
The concerns never name a product. For each cloud we write a binding: which service does each job. This page is the AWS binding, with a tyre manufacturer as the example. On average, by our judgement, about a third of each concern comes from AWS (36%), nearly half is shared code we write once (45%), and about a fifth is what each agent team writes (19%).
Already seen the Azure version? Compare the harness on Azure and the Azure tech stack, concern by concern. Why we work from specs: spec-driven development.
The big picture
One request, top to bottom, on AWS
Read it as one request: a dealer or an engineer asks, through a channel, past the front door, to one vertical agent, which uses the shared assets to reach the company's systems. The left rail is the life of every agent; the right rail is governance.
| Band | What it does on AWS |
|---|---|
| End users | Dealers, fleet customers and consumers sign in through Cognito or the existing dealer portal; plant, quality, sales and service staff through IAM Identity Center. |
| Channels | Web and app chat, WhatsApp and SMS through AWS End User Messaging, email through SES, and the contact centre through Amazon Connect. |
| Front door | CloudFront and WAF at the edge; Bedrock Guardrails for hidden instructions; Comprehend to find personal data; our router rules; API Gateway limits. |
| Orchestration | Our router picks one agent; agents hand work to each other over A2A; a registry in Git names the owner and version of each. |
| Vertical agents | One ECS service per business job. Each has an autonomy level: answers, suggests, drafts for a person, or acts after checks. |
| Horizontal assets | Twelve shared parts every agent reuses (next sections). |
| Systems of record | SAP S/4HANA, the dealer management system, MES, the plant historian and IoT SiteWise, quality and LIMS, warranty, CRM and the supplier portal, reached only through MCP tools with the user's own rights. |
End to end
The design, from account to release
This is a reference architecture, so every choice below is made. The tyre maker is only the example. Read the drawing first, then follow one request through it, then the nine design tiers (H0 to H8).
One request, step by step
| # | Where | What happens |
|---|---|---|
| 1 | Dealer | A dealer in Pune asks in the portal chat: "Can I get 40 of 205/55 R16 by Friday, and at what price?" The portal's sign-in gives a token (Cognito, federated). |
| 2 | Edge | CloudFront and WAF filter the request; the load balancer passes it to the router on ECS. |
| 3 | Router | Checks the token, runs the input guardrail (ApplyGuardrail), opens the turn in PostgreSQL (one turn per conversation) and picks the dealer assistant. |
| 4 | Context | The assembler builds the turn: recent conversation from PostgreSQL, the dealer's own price list passages from pgvector. |
| 5 | Model | The loop calls the model through AgentCore Gateway to Claude on Bedrock in Mumbai; the gateway checks the allow-list and the token budget. |
| 6 | Tools | The model asks for check_stock and get_price. Gating checks the arguments and the dealer scope; our issuer mints a 5-minute token on the dealer's behalf; the SAP MCP server calls SAP over Direct Connect. |
| 7 | Way out | Our rules check the reply: the price comes from this dealer's list, the date from SAP's confirmed stock. The grounding check confirms nothing was invented. |
| 8 | Reply | The dealer gets stock, price and date. "Place the order" would start an approval: a DBOS wait and a card to the sales desk. |
| 9 | Trace | Every step was an OpenTelemetry span sent to the ADOT sidecar, then on to X-Ray and CloudWatch; the model call is logged to S3; the meter wrote the cost of the turn. |
H0 · Foundation
The plot of land: accounts, region, private network, keys, and the code that builds it all.
| Decision | We use | Not used | Why |
|---|---|---|---|
| Accounts | AWS Organizations + Control Tower: agents-dev, agents-test, agents-prod, plus shared security, log-archive, network and shared-services accounts | one account for everything | Agents face the internet; a problem in one account cannot reach another. The data platform has its own accounts in the same organisation. |
| Region | Mumbai (ap-south-1) primary, Hyderabad (ap-south-2) for disaster recovery; a policy (SCP) blocks every other region except us-east-1 for global services | active-active across regions | Indian personal data stays in India. A Frankfurt copy is the same Terraform with another region, built only when EU personal data arrives. |
| Network | One VPC per account across 2 zones, private subnets only; the load balancer is the only public part; Transit Gateway in the network account; outbound only through Network Firewall with a domain allow-list | open NAT egress | One way in, one way out, both inspected. |
| Private paths | VPC endpoints for Bedrock, AgentCore, STS, KMS, Secrets Manager, CloudWatch Logs, X-Ray, ECR, SQS, AppConfig; S3 gateway endpoint | public AWS endpoints | No call to the model or to a secret crosses the internet. |
| Keys | KMS customer keys per kind of data: agent state, logs and transcripts, knowledge | one key for all | Erasing or rotating one kind of data leaves the others alone. |
| Blueprint | Terraform for everything, shared modules with the data platform | AWS CDK | One tool for AWS and Databricks (it has a Terraform provider). |
H1 · Model door
Every model call goes through one governed door, so models can be swapped and spend can be capped.
| Decision | We use | Not used | Why |
|---|---|---|---|
| Gateway | Bedrock AgentCore Gateway in front of Bedrock; LiteLLM on Fargate as the fallback behind the same port | agents calling Bedrock directly | The counterpart of APIM's AI gateway: identity, limits, fallback and logging in one place. The agent never holds a model key. |
| Models | Claude on Bedrock: Sonnet for reasoning, Haiku for routing and short jobs; the model per job lives in AppConfig. The reference implementation's dev setup uses a free open-weight model (Groq) behind the same gateway | Intelligent Prompt Routing | Picking the model is a decision we test offline with the golden set, not one the platform makes per prompt. |
| Residency | Calls carrying Indian personal data run in Mumbai; no Asia-Pacific cross-region profile for them | APAC cross-region inference for personal data | That profile can route to Tokyo, Singapore or Sydney. |
| Spend | One application inference profile per agent (tagged), a token budget per turn, prompt caching on the fixed part of the context | — | Cost per agent comes from AWS; cost per business case comes from our meter (H7). |
H2 · Runtime and loop
Where our code runs, and the loop that plans, acts and checks.
| Decision | We use | Not used | Why |
|---|---|---|---|
| Compute | ECS on Fargate: one service for the router, one per agent, one per MCP tool server | AgentCore Runtime, EKS, Lambda | No cluster to run, the closest match to Container Apps. AgentCore Runtime gets a pilot later; Lambda's cold starts and limits fit a 60-second turn badly. |
| Loop | Our hand-written loop (about 400 lines): 12 steps, 60 seconds and a cost cap per turn; Pydantic AI only as the model layer | Bedrock Agents, Strands, LangGraph | The loop is where every check has an exact position; it stays ours on every cloud. |
| Model layer | Pydantic AI under our loop: typed calls, structured output and one interface for Bedrock and Groq | Claude Agent SDK, raw boto3 Converse | The Agent SDK is Claude-only and runs its own loop out of sight. Calling boto3 directly would mean rebuilding typing, retries and output checks. Pydantic AI helps without taking over the loop. |
| Front | Production: CloudFront + WAF → Application Load Balancer. Dev: API Gateway HTTP API, with the channel's webhook answered before the turn runs | — | API Gateway caps a request at 30 seconds, below the 60-second turn. |
| Flow | SQS FIFO with one message group per conversation; one turn per conversation at a time; priority classes | — | Order is kept per conversation; a plant breakdown goes before a marketing question. |
| Registry | A catalogue file in Git: agent, version, owner, autonomy level | AWS Agent Registry | Not offered in Mumbai; Git is enough. |
H3 · State, memory, knowledge
What the agent keeps, for how long, for whom, and where it finds sourced answers.
| Decision | We use | Not used | Why |
|---|---|---|---|
| Database | Amazon RDS for PostgreSQL, Multi-AZ: one instance, schemas agent_state, dbos and knowledge; one database role per app | Aurora Serverless v2, DynamoDB | PostgreSQL, the same engine as on Azure. Serverless v2 never pauses while DBOS holds a connection, so plain RDS is cheaper and simpler. |
| Memory | Conversation and case tables; retention per data class; erase per person (DPDP Act) | AgentCore Memory | Memory must follow our retention and erasure rules. |
| Knowledge | pgvector in the same database, Bedrock embeddings; the entitlement filter is a SQL WHERE clause | Bedrock Knowledge Bases, OpenSearch | A dealer sees only their own price list and orders; one store, one backup, one access model. |
| When to outgrow it | Stay on pgvector (HNSW index) up to roughly 10 million passages or when searches start slowing turns; then move knowledge to OpenSearch Serverless or S3 Vectors behind the same retrieval port | — | State and memory are ordinary rows and stay in PostgreSQL; only knowledge search would move. |
| Recovery | Automated backups + AWS Backup; a read replica in Hyderabad, promoted in a disaster (back within 4 hours, at most 15 minutes lost) | Aurora Global Database | Enough for the stated targets at a fraction of the cost. |
H4 · Tools to systems of record
How agents reach SAP, the dealer system, the plants and the data platform, with the user's own rights.
| Decision | We use | Not used | Why |
|---|---|---|---|
| Tool servers | One MCP server per system on Fargate: SAP S/4HANA, dealer management, MES and historian, warranty, CRM. Read tools first; write tools only behind an approval | agents calling systems directly | The protocol is open, so the servers move clouds unchanged. |
| Gating | Before every call: argument check (JSON Schema), scope check (this dealer's records only), idempotency key, size cap | — | Protects orders, prices and warranties; ours on every cloud. |
| On behalf of | Our token issuer mints a 5-minute token for the far system on behalf of the user, signed with a KMS key; tool servers check it against our published keys (JWKS) | AgentCore Identity | The same pattern as the Azure FNOL agent; Cognito has no on-behalf-of exchange. |
| SAP | SAP's OData APIs over Direct Connect or VPN through Transit Gateway | reading SAP's database | SAP's own rules and locks stay in force. |
| Data platform | One MCP tool over a Databricks SQL warehouse (Metric Views) through PrivateLink, with the user's Unity Catalog rights | reading raw S3 files | Numbers come from the governed semantic layer. |
H5 · Identity, privacy, guardrails
Who is asking, what data the turn may use, and the checks on the way in and out.
| Decision | We use | Not used | Why |
|---|---|---|---|
| People | Cognito user pool for dealers, fleets and consumers, federating the dealer portal over OIDC; staff through IAM Identity Center | — | Dealers keep one login; the harness only needs a verified token. |
| Services | IAM task roles; secrets in Secrets Manager encrypted with KMS | keys in code or environment | Nothing to leak. |
| Way in | Bedrock Guardrails through the ApplyGuardrail API (prompt attacks, denied topics) on user input and on tool results | NeMo Guardrails | One detector port; works with any model. |
| Way out | Our 30 positioned rules (no price or warranty outside policy, no order changed without OK, a dealer's own records only) + contextual grounding check | — | Every rule that protects money or a customer is ours. |
| Personal data | Comprehend PII plus our recognisers for Aadhaar, PAN, GSTIN and vehicle numbers, redacting before telemetry; Macie on the buckets | — | Telemetry and logs never carry raw personal data. |
H6 · Channels and the human desk
How people reach the agent, and how a person takes over or approves.
| Decision | We use | Not used | Why |
|---|---|---|---|
| Web chat | Our chat page inside the dealer portal | — | Simple, and the portal already has the sign-in. |
| Other channels | WhatsApp and SMS through AWS End User Messaging; email through SES; one conversation id across channels | — | Dealers in India live on WhatsApp. |
| Desk | Chatwoot on Fargate as the human desk; Amazon Connect adapter when a contact centre exists | building our own desk | A desk with the full history, without building one. |
| Approvals | DBOS durable waits in the same PostgreSQL; an approval card to the sales or warranty desk; autonomy level per agent | Step Functions task tokens | Survives restarts, same code as on Azure, no extra service. |
H7 · Observability, replay, cost
What we can see, prove and pay for. Everything goes through OpenTelemetry.
| Decision | We use | Not used | Why |
|---|---|---|---|
| Telemetry standard | OpenTelemetry SDK in our code, one helper, GenAI semantic conventions; personal data redacted before export | AWS SDK calls for telemetry | The code never changes between clouds; only the collector's config does. |
| Collector | AWS Distro for OpenTelemetry (ADOT) as a sidecar in every ECS task: traces to X-Ray, metrics and logs to CloudWatch | — | AWS's supported build of the OpenTelemetry collector. |
| Dashboards and alerts | CloudWatch Application Signals for SLOs; alarms as code (Terraform) to SNS email and chat | — | Turn latency at the 95th percentile, error rate, guardrail blocks, cost per hour. |
| Evidence | Bedrock model invocation logging to S3 in the log-archive account (1 year); CloudTrail for who changed what | — | Answers an auditor's question months later. |
| Replay | Run record + cassettes + config fingerprint in S3 | — | "Why did the agent tell this dealer that price?" needs a replay; no cloud offers one. |
| Cost | Inference profile tags + Budgets + CUR 2.0; our meter per turn and per case with a hard budget stop | — | Cost per warranty claim triaged, not just per agent. |
H8 · Evaluation and release
How a change is proven safe and rolled out, and rolled back.
| Decision | We use | Not used | Why |
|---|---|---|---|
| CI | GitHub Actions with OIDC into AWS (no stored keys); four gates: AgentTwin world, golden sets, red team (PyRIT, promptfoo), cost and latency | CodePipeline | The same gates as on Azure. |
| Images | Amazon ECR, scanned on push | public registries in production | Private images, close to Fargate. |
| Release | One release bundle (prompt + model + policy + config fingerprint); ECS blue/green through CodeDeploy with alarms that roll back | — | Prompt, model and policy change together and roll back together. |
| Runtime config | AWS AppConfig: kill switch per agent, limits, model per job, read before every turn | — | A bad behaviour is switched off in seconds, without a deploy. |
| Online evaluation | Each night a sample of live turns is graded; Bedrock evaluation is one judge plug-in; misses go to the golden set | — | The golden set keeps learning from production. |
Every concern, bound to AWS
The 22 concerns, one table
For each concern: what AWS offers and whether we use it, the open-source options, and what we code once for every agent, and why.
use in the design planned next phase considered not picked, or newer service to check not used and why
| Concern | On AWS | Open source | Ours, built once, and why |
|---|---|---|---|
| Think · how the agent decides | |||
| L1 · Context assembly What the model sees each turn, built in one place, within a budget. |
|
| Our assembler: cache-friendly order, fencing, trimming from the middle, transcript check; prompts in Git No product on any cloud assembles a turn the way a regulated agent needs, so we code it once. |
| L2 · Model invocation Every model call goes through one door. |
|
| Our gateway policy: allow-list per job, token budget, one retry budget, token metric per agent AgentCore Gateway is AWS's AI gateway, the same job APIM's AI gateway does on Azure: one governed door with identity, rate and token limits, fallback and logging. Our policy (allow-list per job, budgets) sits on top as config. Check region availability and token-limit features before go-live. The agent never holds a provider key. |
| L4 · Control loop Plan, act, check, with step and no-progress limits. |
|
| Hand-written loop, about 400 lines; budgets of 12 steps, 60 s and a cost cap per turn Agents that touch orders, warranties and money need every step testable and checks at exact positions. Our own loop runs on any model and any cloud. |
| L5 · State & memory What is kept, for how long, and for whom. |
|
| 5 agent tables + DBOS state in the same database; retention and erase per person Plain RDS PostgreSQL, the same engine as on Azure. One managed store that also holds the approval engine and, later, pgvector: no extra infrastructure. |
| 18 · Knowledge & retrieval Finds sourced passages, filtered by what this user may see. |
|
| Entitlement filter as SQL: a plant engineer sees machine manuals, a dealer sees only their own orders and price lists One store, one backup, one access model; the entitlement check is just a SQL filter. |
| Act · how it reaches people and systems | |||
| CH · Channels & human desk How dealers, customers and staff reach the agent, and how a person takes over with the full history. |
|
| Our chat page and desk API; one conversation id across channels If the company already runs a contact centre, hand off into it (Connect). Otherwise Chatwoot gives the desk without building one. |
| L3 · Tool layer Actions through typed tools, checked before they run. |
|
| Gating before every call, idempotency keys, size caps, argument checks with jsonschema The tool protocol is open (MCP), so the servers move clouds unchanged. The gating that protects orders, prices and warranties is ours. |
| L6 · I/O contracts Typed inputs and outputs at every edge. |
|
| Frozen Pydantic models everywhere; MCP structured output Contracts are code, not cloud. SAP and dealer-system shapes are typed once at their adapters. |
| L8 · Concurrency & flow Who gets served first when load spikes. |
|
| One turn per conversation (ledger row), priority classes (plant breakdowns before marketing), honest shedding AWS gives the queue and the limits. Which request waits, and what we tell the person, is ours. |
| 17 · Agent-to-agent Agents hand work to agents through typed, bounded, traced hand-offs. |
|
| Registry in Git: who, which version, owner, autonomy level A2A is an open protocol, so hand-offs work across clouds. The registry stays a file we own. |
| PS · Performance & scalability Fast enough at the 95th percentile; more servers when load rises; the model quota is the real limit. |
|
| Per-turn time budget (60 s), cache-friendly context order Servers are stateless and scale out; the model quota is what to plan and load-test. |
| Control · what keeps it safe | |||
| L16 · Identity & authz Who is asking, and whose authority the agent acts with. |
|
| Principal → dealer, customer or staff identity; on-behalf-of: the agent acts with the user's rights, never its own Checking a token is local and cloud-neutral. Ports make leaving Cognito a configuration change. |
| PV · Privacy & data protection Personal data of dealers, fleet drivers and consumers used only for this case, kept in its country, masked, and forgotten on request. |
|
| redact() in telemetry (Aadhaar, PAN, phone, vehicle number), no_pii_echo rule, forget() + retention per the DPDP Act and GDPR Residency and encryption come from AWS. Which data a turn may use, masking and erasure are domain rules we code once. |
| L7 · Policy enforcement Guardrails on the way in and on the way out. |
|
| Our positioned rules (30 checks): no price or warranty promised outside policy, no order changed without OK, a dealer's own records only We own every position and every rule; every check that protects money or a customer is ours. AWS supplies detectors behind one port. |
| L10 · Failure handling Typed failures, fallbacks, undo paths, honest messages. |
|
| Typed failures, ledger abandon-and-retry, an honest message instead of a guess Retries without a budget hide outages. One retry budget per turn is ours on any cloud. |
| L14 · Human-in-the-loop Approvals and hand-offs the agent cannot grant itself. |
|
| Approval and escalation tables, desk API, approval cards to staff A warranty or order approval must survive restarts. DBOS keeps it in the database we already run, so it moves clouds unchanged. |
| AV · Availability & disaster recovery A lost server or zone goes unnoticed; a lost region is rebuilt from code and backups. |
|
| Stateless servers + durable state; a rebuild drill Two Indian regions make a DR pair without data leaving the country. Target: back within 4 hours, at most 15 minutes of data lost. |
| See · what we can see and prove | |||
| L11 · Observability Traces, metrics and alerts for every step. |
|
| One telemetry helper on the OpenTelemetry SDK: spans per step with GenAI attributes, personal data redacted before export, alert rules as code OpenTelemetry is the one standard from code to collector: the code never imports an AWS SDK for telemetry. On Azure the same spans go to Application Insights; on AWS the ADOT collector sends them to X-Ray and CloudWatch. Moving clouds changes the collector's config file, not the code. |
| L9 · Determinism & replay A past decision can be re-run and explained. |
|
| Run record + cassettes + config fingerprint "Why did the agent say that to this dealer?" needs a replay. No cloud offers it, so it is ours. |
| L13 · Cost accounting Spend measured per unit of business work. |
|
| Meter per turn and per case (cost per warranty claim triaged), hard budget stop AWS tells you cost per agent; only we know cost per warranty claim or per dealer order. |
| Change · how it changes safely | |||
| L12 · Eval harness Golden sets and graders that gate every release. |
|
| Golden sets owned by quality, warranty and dealer-service leads, hard graders, 4 CI gates, model selection run The pretend dealer network and plant (AgentTwin) and the hard graders are the same on any cloud; AWS judges are one plug-in. |
| L15 · Release & config Prompt, model and policy versioned and rolled back together. |
|
| One release bundle: prompt + model + policy + config fingerprint AppConfig is a close match for App Configuration; the bundle and rollback rule are ours. |
Services marked "check current status" are newer AWS offerings; confirm their availability in Mumbai and Hyderabad (and Frankfurt, if the EU copy is built) before relying on them.
Who builds what
What AWS gives, what we build, what each agent writes
For each concern, how much does AWS give us, how much do we code once for every agent, and how much does each agent team still write? AWS gives the plumbing. The parts that make an agent safe for dealers and customers stay in our code.
The rule that keeps this healthy: when two agents write the same code, it moves into the shared middle (blue). When AWS catches up, it moves to the left (grey).
The five big choices
Decided, with the options we dropped
| Decision | Options | We chose | Verdict | Why |
|---|---|---|---|---|
| Which model? | Claude on Bedrock · Llama or Nova on Bedrock · a model outside AWS via the gateway | Claude on Bedrock as the default for every job, behind our gateway. The golden set picks the model per job offline; cheaper models win where they pass. | Pursue: Claude on Bedrock | Data stays in the AWS account; switching model is one line in AppConfig. |
| Which gateway (the one door to models)? | Bedrock AgentCore Gateway · LiteLLM proxy on ECS Fargate · API Gateway + Lambda (build it ourselves) | AgentCore Gateway, with our policy (allow-list, budgets) as its rules; LiteLLM on Fargate kept as the fallback behind the same port. | Pursue: AgentCore Gateway | Managed by AWS, the closest match to APIM's AI gateway, and it also fronts MCP tools. LiteLLM only if a rule we need is missing or a non-AWS model is required. Check Mumbai availability first. |
| Where does our code run? | ECS Fargate · Bedrock AgentCore Runtime · EKS | ECS Fargate: the closest match to Azure Container Apps, no cluster to run. | Pursue: ECS Fargate | AgentCore Runtime is worth a pilot later; EKS only if the company already runs Kubernetes. |
| How do approvals wait? | DBOS on RDS PostgreSQL · Step Functions task tokens | DBOS: the same code as on Azure, state in the PostgreSQL we already run. | Pursue: DBOS | Step Functions works, but it is one more service and AWS-only. |
| How do dealers and customers sign in? | Amazon Cognito · federate the existing dealer portal over OIDC | Cognito user pool, federating the dealer portal over OIDC. | Pursue: Cognito + portal federation | Dealers keep one login; the harness only needs a verified token. |
Moving clouds
Azure to AWS in one table
| Job | On Azure | On AWS |
|---|---|---|
| Front door | Azure Front Door + WAF | CloudFront + AWS WAF |
| AI gateway | API Management (AI gateway) | Bedrock AgentCore Gateway (LiteLLM on Fargate as fallback) |
| Models | Foundry Models | Amazon Bedrock (Converse API) |
| Where code runs | Azure Container Apps | Amazon ECS on Fargate |
| Sign-in | Entra ID, External ID | IAM Identity Center (staff), Cognito or portal OIDC (dealers, fleets, consumers) |
| Secrets and keys | Key Vault, managed identity | Secrets Manager + KMS, IAM task roles |
| Guardrails | AI Content Safety, Prompt Shields | Bedrock Guardrails (ApplyGuardrail) |
| Finding personal data | AI Language PII | Comprehend PII, Macie for S3 |
| Database | Azure Database for PostgreSQL | Amazon RDS for PostgreSQL |
| Queue | Service Bus | SQS FIFO |
| Traces and metrics | OpenTelemetry → Application Insights | OpenTelemetry → ADOT collector → X-Ray + CloudWatch |
| Runtime config, kill switch | App Configuration | AWS AppConfig |
| Cost | Cost Management, APIM token metric | Budgets, Cost Explorer, Bedrock inference-profile tags |
| Evaluation | Foundry evaluation, AI Red Teaming Agent | Bedrock evaluations, PyRIT, promptfoo |
| Infrastructure as code | Bicep + azd | Terraform or AWS CDK |
| Contact centre | Communication Services, Dynamics | Amazon Connect, End User Messaging, SES |
What does not change: specs, tests and agent code.
- The 22 concerns and the specs (AI Harness Catalog, AI Assurance Catalog, agent specs)
- The agent code: router, loop, context assembler, tools, approval flows
- The tests: AgentTwin worlds, golden sets, hard graders, the four CI gates
- The rules: every guardrail position and every check that protects money or a customer
- The ports: each AWS service sits behind the same interface the Azure one did
That is the point of binding through ports: moving from Azure to AWS is a new binding file and a set of adapters, not a rebuild.
Finally
What we are using
The whole binding on one page: for each concern, only what we picked. AWS services and open-source tools marked planned are chosen but not yet switched on; everything else in the tables above was considered and left out.
| Concern | AWS service | Open source | Our own code |
|---|---|---|---|
| Think | |||
| L1 · Context assembly | — | Jinja templates | Our assembler: cache-friendly order, fencing, trimming from the middle, transcript check; prompts in Git |
| L2 · Model invocation | Amazon Bedrock, Converse API Cross-region inference profiles Bedrock AgentCore Gateway, inference targets Bedrock application inference profiles | Pydantic AI | Our gateway policy: allow-list per job, token budget, one retry budget, token metric per agent |
| L4 · Control loop | — | — | Hand-written loop, about 400 lines; budgets of 12 steps, 60 s and a cost cap per turn |
| L5 · State & memory | Amazon RDS for PostgreSQL, Multi-AZ | psycopg + pool | 5 agent tables + DBOS state in the same database; retention and erase per person |
| 18 · Knowledge & retrieval | pgvector on the same RDS PostgreSQL (planned) | pgvector (planned) | Entitlement filter as SQL: a plant engineer sees machine manuals, a dealer sees only their own orders and price lists |
| Act | |||
| CH · Channels & human desk | AWS End User Messaging (planned) Amazon SES (planned) Amazon Connect (planned) | Chatwoot (planned) | Our chat page and desk API; one conversation id across channels |
| L3 · Tool layer | MCP servers on ECS Fargate or Lambda Bedrock AgentCore Gateway (planned) | MCP Python SDK | Gating before every call, idempotency keys, size caps, argument checks with jsonschema |
| L6 · I/O contracts | API Gateway request validation (planned) | Pydantic JSON Schema SAP OData / BAPI shapes (planned) | Frozen Pydantic models everywhere; MCP structured output |
| L8 · Concurrency & flow | Amazon SQS FIFO (planned) API Gateway usage plans Bedrock quotas | — | One turn per conversation (ledger row), priority classes (plant breakdowns before marketing), honest shedding |
| 17 · Agent-to-agent | A2A over HTTPS behind API Gateway (planned) | A2A SDK (planned) | Registry in Git: who, which version, owner, autonomy level |
| PS · Performance & scalability | ECS Fargate service auto scaling Bedrock prompt caching Distributed Load Testing on AWS (planned) | — | Per-turn time budget (60 s), cache-friendly context order |
| Control | |||
| L16 · Identity & authz | Amazon Cognito (planned) IAM Identity Center IAM roles for tasks Secrets Manager + KMS | PyJWT | Principal → dealer, customer or staff identity; on-behalf-of: the agent acts with the user's rights, never its own |
| PV · Privacy & data protection | Comprehend PII (planned) Amazon Macie (planned) Mumbai (Hyderabad for DR) Frankfurt (planned) KMS, PrivateLink | — | redact() in telemetry (Aadhaar, PAN, phone, vehicle number), no_pii_echo rule, forget() + retention per the DPDP Act and GDPR |
| L7 · Policy enforcement | Bedrock Guardrails ApplyGuardrail API Contextual grounding check (planned) | Llama Prompt Guard 2 (planned) | Our positioned rules (30 checks): no price or warranty promised outside policy, no order changed without OK, a dealer's own records only |
| L10 · Failure handling | Cross-region inference ALB / ECS health checks | Own resilience wrapper | Typed failures, ledger abandon-and-retry, an honest message instead of a guess |
| L14 · Human-in-the-loop | Amazon Connect (planned) | DBOS | Approval and escalation tables, desk API, approval cards to staff |
| AV · Availability & disaster recovery | Multi-AZ ECS + RDS Multi-AZ RDS cross-region read replica (planned) Route 53 health checks (planned) AWS Backup (planned) | Terraform or AWS CDK | Stateless servers + durable state; a rebuild drill |
| See | |||
| L11 · Observability | AWS Distro for OpenTelemetry (ADOT) collector CloudWatch + Application Signals AWS X-Ray Bedrock model invocation logging Managed Prometheus + Grafana (planned) | OpenTelemetry SDK + GenAI semantic conventions | One telemetry helper on the OpenTelemetry SDK: spans per step with GenAI attributes, personal data redacted before export, alert rules as code |
| L9 · Determinism & replay | — | Own recorder | Run record + cassettes + config fingerprint |
| L13 · Cost accounting | Bedrock application inference profiles AWS Budgets Cost Explorer CUR 2.0 data exports (planned) | Own meter | Meter per turn and per case (cost per warranty claim triaged), hard budget stop |
| Change | |||
| L12 · Eval harness | Bedrock model evaluation (planned) Bedrock RAG evaluation (planned) AgentCore Evaluations (planned) | AgentTwin, pytest, pydantic-evals promptfoo, PyRIT (planned) | Golden sets owned by quality, warranty and dealer-service leads, hard graders, 4 CI gates, model selection run |
| L15 · Release & config | AWS AppConfig ECS blue/green deployments Amazon ECR | GitHub Actions + OIDC | One release bundle: prompt + model + policy + config fingerprint |
How the code fits
Library dependencies
Agents hold only their domain code. Everything else comes from two libraries we own and a handful of open-source ones. Moving to AWS adds one install option, [aws], to each library: new adapters behind the same ports, nothing else.
AgentCore, part by part
Which AgentCore parts we use
AgentCore is AWS's set of building blocks for agents. We take the parts that are plumbing and keep the parts that make our agents ours.
| AgentCore part | Verdict | Why | What of ours sits there |
|---|---|---|---|
| Gateway | Use | The one door to models and MCP tools: any provider, fallback, request and token limits per user, guardrails at the door. | Our gateway policy (allow-list per job, budgets) rides on it |
| Policy | Use | Cedar rules on who may call which tool; Bedrock Guardrails at the gateway for detector-type checks. | Rules from our guardrail catalog that are detector-shaped |
| Observability | Use | Shows our OpenTelemetry traces in CloudWatch; nothing to install in our code. | Our telemetry helper + ADOT |
| Evaluations | Use, as a plug-in | One more evaluator provider inside our eval framework, including online evaluation of live traces. | evalharness decides what runs where |
| Identity | Not now | Token vault and OAuth to third-party tools. Add it when a SaaS tool needs a user's consent. | Our on-behalf-of token issuer for SAP and the dealer system |
| Memory | No | Our memory must follow our retention and erasure rules and sit next to the approvals. | Memory tables in RDS PostgreSQL |
| Runtime | Not now | Managed hosting for agent containers; worth a pilot later. | ECS on Fargate |
| Managed harness | No | AWS runs the loop, memory and evaluation for you. | Our loop, the part we keep on every cloud |
| Code Interpreter, Browser | No | No tyre-company job needs them yet. | — |
| Agent Registry | No | Not offered in Mumbai. | Catalogue file in Git |
Reuse, don't rebuild
What we already built, and how it lands on AWS
| Asset | What it is | On AWS |
|---|---|---|
| Agent harness library (agent_harness) | Loop, context assembler, tool gating, DBOS approvals, telemetry helper, ports and adapters, the four offline gates with AgentTwin | Unchanged. Add the AWS adapters (model, state, identity, secrets, config, telemetry); each passes the contract tests the Azure ones pass. Offline gates run in GitHub Actions and need no cloud. |
| Guardrail catalog | Every check, with its id, what it catches and where it may run | The catalog stays the source of truth. Each rule is bound to a place: our code inside the loop (most rules), Bedrock Guardrails at the gateway for detector-type checks (prompt attack, harmful content, PII), the eval framework for online and release checks. Same rule id everywhere, so one report covers all places. |
| Eval framework (evalharness) | Evaluator, TraceSource, Judge and ResultSink ports; domain patterns; cost rules; five positions | Add AWS adapters: a trace source that reads OpenTelemetry spans from CloudWatch, Bedrock Guardrails and AgentCore Evaluations as evaluator providers, Bedrock as a judge model, and a result sink that writes scores back as OpenTelemetry evaluation events to CloudWatch. Nightly online run as a scheduled ECS task. |
| Trace viewer (the LangSmith job) | We never built our own; local dev uses self-hosted Langfuse | Production: CloudWatch generative AI observability, fed by the same ADOT pipeline, with our scores written next to the traces. Langfuse on ECS is an optional second destination for teams that want prompt playground and annotation queues; adding it is a collector config change. |
AWS's gateway gives the door: any model, fallback, rate and token limits per user, guardrails at the door. It does not give spend in money per turn or per business case, a retry budget per turn, rules at exact positions inside the loop, or replay. Those stay ours.
From design to running code
The reference implementation
The same path the motor-claims FNOL agent took on Azure: write the binding, write the adapters, add a configuration overlay, build the infrastructure, run it, then add the next agent. The agent's code and tests do not change.
| Step | What | How | State |
|---|---|---|---|
| R0 | Binding written down | clean-ai-engineering/stacks/aws.yaml: every port named with its AWS adapter | done on paper (19 Oct) |
| R1 | AWS adapters | In the agent_harness library: agentcore-gateway, rds-postgresql, cognito + our token issuer, secrets-manager, appconfig, otel → ADOT; a contract test per port, the same tests the Azure adapters pass | done (3 Nov), tested offline; live recordings wait for an AWS account |
| R2 | The FNOL agent on AWS, unchanged | Add config/aws.yaml to the motor-claims FNOL agent: a new overlay, no change to the agent's code. This proves that only the binding changes | done (3 Nov): one profile file, two overlays, keys published at the edge; no agent code changed |
| R3 | Infrastructure as code | Terraform for H0 to H8 at dev size (one account, budget alarm), checked offline first | planned |
| R4 | Running on AWS | Deploy to AWS dev; first claim end to end, traced in X-Ray through ADOT | planned |
| R5 | Tyre dealer assistant | The second agent on the same stack: SAP and dealer-system MCP servers against an AgentTwin dealer world | planned |
Read next
Go deeper
Agent harness tutorial (every concern explained, on the Azure example) · Spec-driven development · Data platform architecture · Guardrail catalog · AgentTwin.
To walk through this with your team, write to basant@cleandataengineering.ai.