Domain 3: Integration
Integration is where Claude meets the rest of the organisation: tools, identities, data stores, monitoring and other agents. The questions give you a working system with one gap (too much access, too slow, impossible to debug, retrieving the wrong thing, wired up the wrong way) and ask which configuration or mechanism closes it with the least added complexity. It is the heaviest domain, so expect about twelve of the 63 questions.
19% of the exam8 guide lines · 16 practice questionsGuide v1.0
Other domains: D1 · D2 · D3 · D4 · D5 · D6 · D7
3.1 · Evaluate tool/agent configuration for capability bloat
Guide line: “Evaluate tool/agent configuration for capability bloat”
Capability bloat means an agent has been given more tools, permissions or data access than its role uses. It usually happens by accident: a team connects a whole MCP server or reuses an admin account because it is quicker.
Each extra capability is attack surface: something a prompt injection, a confused model or a misuse could trigger. It also makes the model's job harder, because more tools means more chances to pick the wrong one.
The fix is least privilege: list what the role actually does, and remove everything else. Removing a capability eliminates the risk; logging it or asking for confirmation only reduces or reveals it. Apply it in two places: the tool list the agent sees, and the identity (account or token) the tools run under, so that even a tool that slips through cannot do more than the role allows.
Know this
- Least privilege: an agent gets the minimum tools and permissions its role needs.
- Removing a capability is a preventive control; logging and confirmation prompts are detective or compensating controls.
- Read-only and write tools should be separate, so a read-only role never holds write access.
- Fewer, well-described tools also improve tool selection.
- Restrict both the tools the agent can see and the permissions of the account those tools use.
- A bigger or better model does not change what the agent is authorised to do.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| The role never uses a tool the agent has | Remove the tool | Eliminates the risk entirely |
| The role needs the tool, but rarely, and the action is high-impact | Keep it behind human approval | Removing it would block real work |
| The role needs a tool, but only part of it | Replace with a narrower tool | Keeps the job possible with less power |
| The agent runs under a shared or admin account | Give it its own scoped identity | The permission layer is the backstop if a tool slips through |
| Auditors need a record of what happened | Add logging as well | Logging supports, it never replaces, least privilege |
Worked example
Situation: An internal HR assistant was connected to the full HR MCP server, which can read, edit and delete employee records. Staff only ask it policy questions and look up their own leave balance.
Decision: Give it a read-only policy search tool and a leave-balance tool scoped to the signed-in employee. Remove edit and delete entirely.
Why: Nothing in the role needs write access, so removing it closes the risk. Approval prompts on edits would still leave the capability one injected instruction away.
Practice
A finance team's reporting agent reads ledger data and drafts monthly summaries. It was set up with the same service account as the payments system, which can also create and approve payments. The agent has never needed to move money. What should the architect change first?
- Add an alert that notifies finance whenever the agent creates a payment.
- Give the agent its own read-only account limited to ledger data.
- Require a human to confirm any payment the agent tries to approve.
- Add a system-prompt rule that the agent must never create payments.
Show answer
Answer: B. The deciding fact: the agent has never needed to move money. Least privilege removes the payment capability by giving it a read-only identity.
- A ❌ Works, but it only detects misuse after the payment exists.
- B ✅ Give the agent its own read-only account limited to ledger data.
- C ❌ Works, but it keeps a capability the role never uses; it would fit if the role sometimes needed payments.
- D ❌ Works, but a prompt rule is probabilistic and the permission is still there.
A pharma research team's literature agent searches internal papers and summarises them. It is connected to the full document-management MCP server, whose 40 tools include upload, share externally, delete and change permissions, and it runs under an editor account. The agent only ever searches and reads. Which two changes best apply least privilege?
- Log every delete and share call and review the log each week.
- Require a human to approve any delete or external share.
- Allowlist only the search and read tools for this agent.
- Add a system-prompt rule to use search and read tools only.
- Run the agent under its own read-only document account.
Show answer
Answer: C and E. The deciding fact: the agent only ever searches and reads. Least privilege works in two layers: hide the tools the role never uses (which also makes tool choice easier) and narrow the account so nothing that slips through can write.
- A ❌ Works, but logging only reveals misuse after it has happened.
- B ❌ Works, but it keeps capabilities the role never uses; it fits when the role genuinely needs them now and then.
- C ✅ Allowlist only the search and read tools for this agent.
- D ❌ Works, but a prompt rule is probabilistic and every tool is still connected.
- E ✅ Run the agent under its own read-only document account.
3.2 · Analyze authentication and authorization requirements to identify security gaps
Guide line: “Analyze authentication and authorization requirements to identify security gaps”
Authentication answers “who is this?”: the user signs in, or a service proves its identity with a token. Authorization answers “what may they see or do?”: the permissions attached to that identity. Most agent security gaps are authorization gaps: everyone is properly signed in, but the agent reaches data on their behalf with far more access than they have.
The classic gap is the shared service account: the agent calls every system with one powerful identity and decides in its own logic, or in the prompt, whose data to fetch. Any user can then get any data by asking cleverly. This is a confused deputy: a trusted component tricked into using its own authority for someone who lacks it. The fix is delegated access: the agent acts with a token issued for the signed-in user (for example through OAuth), scoped to what that user may do, so the target system enforces the user's own permissions. Remote MCP servers follow the same idea; the MCP specification uses OAuth-based authorization for servers reached over HTTP.
Retrieval needs the same care. If a RAG index is built from documents with different access rules, store each chunk's access-control list (who may read it) as metadata and filter by the user's identity at retrieval time, so restricted text never reaches the model. A system-prompt rule or an output filter applied afterwards is too late: the data has already been read. Finally, keep secrets out of prompts, and log both the user and the agent on every action so audits can tell them apart.
Know this
- Authentication proves identity; authorization decides access. Signing in users does not fix an authorization gap.
- An agent using one shared, all-powerful account lets any user reach any data: act with the user's delegated, scoped token instead.
- Enforce access in the tool or data layer; a prompt instruction is not an access control.
- Permission-aware retrieval filters chunks by the user's access list before they reach the model.
- Remote MCP servers use OAuth-based authorization, so tool calls can carry the user's own scoped access.
- Credentials belong in a secrets store or environment, never in prompts or tool descriptions.
- Audit logs should record which user and which agent performed each action.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| The agent acts for individual users in systems that already have per-user permissions | Delegated, user-scoped token | The system enforces what that user may do |
| Users see answers from documents they could not open themselves | Filter retrieval by each chunk's access list | Restricted text never reaches the model |
| Access rules are coarse and match teams cleanly (say, one index per legal entity) | Separate indexes per group | Simpler, and enough when permissions follow the split |
| A background job with no user behind it (nightly batch) | Its own narrowly scoped service identity | There is no user to delegate from, so least privilege applies to the service |
| Users are not signed in at all | Fix authentication first | Authorization needs a known identity to work from |
Worked example
Situation: An internal assistant answers questions from an index built over the whole company document store, using a crawler account that can read everything. A salesperson asked about bonuses and got figures from an HR compensation file.
Decision: Index each document's access list with its chunks, and filter every search by the signed-in user's groups. Tools that fetch full documents call the store with the user's delegated token.
Why: The gap is authorization, not authentication: the salesperson was properly signed in. A prompt rule like “don't reveal salaries” would still let the model read the file and depends on the model obeying; filtering at retrieval stops the text from ever arriving.
Practice
A consultancy's internal assistant answers questions from a vector index built over every project site. Staff sign in with single sign-on, and site permissions vary document by document. A junior analyst received figures from a board-only document. The system prompt already forbids sharing confidential material. What closes the gap?
- Strengthen the system prompt with examples of content to withhold.
- Add an output classifier that blocks answers about board topics.
- Build a separate index per department and route by the user's team.
- Filter retrieval by each chunk's access list and the user's groups.
Show answer
Answer: D. The deciding fact: permissions vary document by document. Only a per-chunk access filter at retrieval matches that, using the identity single sign-on already provides.
- A ❌ Works, but the model still reads the restricted text and the prompt is not an access control.
- B ❌ Works, but it catches leaks after retrieval and misses ones it doesn't recognise.
- C ❌ Works, but only when permissions follow departments; here they vary per document.
- D ✅ Filter retrieval by each chunk's access list and the user's groups.
An agent drafts replies in employees' mailboxes and books meetings in their calendars. It uses one admin account that can open every mailbox, choosing which mailbox from the name typed in the chat. Employees already sign in through the company identity provider with multi-factor authentication. What is the main gap, and the fix?
- Authorization: act with each signed-in user's own delegated token.
- Authentication: add a second sign-in step before the chat opens.
- Secrets: move the admin credentials into a managed secrets vault.
- Auditing: log which mailbox each request opened and alert on anomalies.
Show answer
Answer: A. The deciding fact: users already authenticate strongly, yet any of them can reach any mailbox. That is an authorization gap, fixed by acting with each user's own delegated permissions.
- A ✅ Authorization: act with each signed-in user's own delegated token.
- B ❌ Works, but sign-in is already strong; the problem is what the agent can reach after it.
- C ❌ Works as hygiene, but the stored account still opens every mailbox.
- D ❌ Works, but it detects misuse rather than preventing it.
3.3 · Evaluate accuracy-latency trade-offs and justify configuration decisions
Guide line: “Evaluate accuracy-latency trade-offs and justify configuration decisions”
A latency SLA (service-level agreement) is a promise such as “95% of replies start within one second”. The p95 is the time 95% of requests beat; it matters more than the average because the slow tail is what users notice. Two timings matter: time to first token (when output starts) and total time (when it finishes).
Almost every accuracy lever costs time: a larger model, extended thinking, retrieving more chunks, a reranker (a second model that re-scores retrieved results), a verification pass, or more agents. Speed levers include a smaller, faster model, prompt caching, running independent calls in parallel, and doing work in advance.
The architect's moves are: route so only hard or high-stakes requests take the slow, accurate path; move checks off the live path (verify asynchronously when the result is not needed instantly); and stream output so a person sees text sooner. Streaming only improves perceived speed for a human reader; it does not help when a program needs the complete result or when the slow part happens before the first word. Justify the choice with a table of accuracy and p95 latency for each candidate configuration on the eval set.
Know this
- Larger models, extended thinking, reranking, more retrieval and verification passes raise accuracy and latency together.
- Judge latency by percentiles (p95, p99) and time to first token, not the average.
- Routing sends only hard or high-risk requests to the slower path, so most traffic stays fast.
- Asynchronous verification fits when the checked result is used later, not in the live reply.
- Streaming improves perceived speed for human readers only; it does not shorten total time.
- Prompt caching cuts time to first token when a long prefix repeats.
- Justify a configuration with measured accuracy and latency from the eval, not intuition.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| Most requests are easy, a few are hard, and the SLA is tight | Route: fast path by default, heavy path for hard cases | Keeps p95 low and accuracy where it matters |
| A check improves accuracy but the result is used later | Verify asynchronously, off the live path | Accuracy gain without user-facing delay |
| A person reads a long answer and complains it feels slow | Stream the output | Perceived speed improves; nothing else changes |
| Every request is high-stakes and the SLA is loose | The heavier, more accurate configuration everywhere | Latency has room, so take the accuracy |
| A long, identical prefix is sent on every call | Prompt caching | Lower time to first token at no accuracy cost |
Worked example
Situation: A bank's internal search assistant must answer within two seconds at p95. Adding a reranker raises answer accuracy from 81% to 90% on the eval but pushes p95 to 3.4 seconds. Accuracy matters most on regulatory questions, which are about 10% of traffic.
Decision: Use the reranker only for questions classified as regulatory; keep the fast path for the rest. Report accuracy and p95 for both paths.
Why: Routing keeps overall p95 inside the SLA while getting the accuracy gain where it counts. Turning the reranker on everywhere breaks the SLA; dropping it everywhere gives up accuracy on the questions that need it.
Practice
A pharmacy's voice agent must start speaking within one second for routine questions; for drug-interaction questions the pharmacy accepts a few seconds because accuracy is critical. A smaller, faster model answers in 600 ms and passes the eval on routine questions but fails often on interactions. The most capable model with a reranker takes three seconds. Which configuration fits?
- The most capable model with reranking, streaming its words to speech.
- The smaller model for every call, with a stronger, more detailed interaction prompt.
- Smaller model for routine calls, heavy path for interaction questions.
- The heavy path for every call, plus a cache of the commonest answers.
Show answer
Answer: C. The deciding fact: the SLA differs by question type. Routing gives routine calls the fast model and sends interaction questions, where slower is acceptable, to the accurate path.
- A ❌ Works, but reranking happens before the first word, so streaming can't bring routine replies under one second.
- B ❌ Works, but the eval already shows the smaller model failing on interactions; a prompt tweak is a guess.
- C ✅ Smaller model for routine calls, heavy path for interaction questions.
- D ❌ Works for repeated questions, but every cache miss still breaks the routine SLA.
An insurer's claims-intake assistant chats with customers live and extracts claim details into a record that adjusters review the next morning. A second model pass that verifies extracted fields raises field accuracy from 94% to 99%, but adds four seconds to each reply and breaks the two-second response SLA. What should the architect do?
- Run verification after the reply, before adjusters open the record.
- Drop the verification pass and accept 94% field accuracy.
- Keep verification inline and stream the reply so text appears sooner.
- Replace the separate pass with extended thinking on the extraction call.
Show answer
Answer: A. The deciding fact: adjusters use the record the next morning, so verification does not need to sit on the live path. Running it asynchronously keeps both the SLA and the accuracy.
- A ✅ Run verification after the reply, before adjusters open the record.
- B ❌ Works for the SLA, but gives up accuracy that is available for free off the live path.
- C ❌ Works when slow output is the problem, but here verification runs before the reply, so the first words are still late.
- D ❌ Works to raise accuracy, but extended thinking also adds latency to the live reply.
3.4 · Analyze observability challenges and select monitoring strategies at scale
Guide line: “Analyze observability challenges and select monitoring strategies at scale”
Observability means being able to tell what a system did and why from the data it records. LLM systems make this hard: the same input can give different outputs, one answer may come from many steps (several model calls, tool calls, retrievals, sometimes several agents), and “wrong” is often a quality judgement rather than an error code.
The backbone is end-to-end tracing: one request ID carried through every step, with each step recorded as a span (a timed record of one operation: its inputs, outputs, model, tokens and duration). Open standards such as OpenTelemetry are commonly used. With a trace you can open one bad answer and see which retrieval returned the wrong chunk or which tool returned an error.
On top of traces, track metrics per request and per step: input and output tokens and cost, latency percentiles (p50, p95, p99, because averages hide the slow tail), tool error rates, and the prompt and model version used. Quality cannot be read from metrics, so run sampled quality evals: score a random slice of live traffic with automated checks or an LLM judge, and alert when scores drop. Handle personal data in logs deliberately (redact or restrict access).
Know this
- One request ID across every agent, model call, tool call and retrieval lets you find the failing step.
- Record tokens and cost per request and per step; daily totals cannot show which feature or step is expensive.
- Use latency percentiles (p95, p99); averages hide slow outliers.
- Quality needs its own signal: sampled evals on live traffic, with alerts on drops.
- Log the prompt version and model version with each request so regressions trace to a change.
- Raising log verbosity without correlation gives more text, not more insight.
- Prompts and responses can contain personal data: redact or restrict access to logs.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| Can't tell which step in a multi-step system caused a bad answer | End-to-end tracing with one request ID | Shows each step's input and output in context |
| Spend rose but volume didn't, or some users are very slow | Per-request, per-step token, cost and latency percentiles | Finds the expensive or slow path that totals and averages hide |
| Quality may be drifting, but nothing errors | Sampled quality evals with alerts | Quality is invisible to system metrics |
| The failing step is already known and needs detail | Richer logging on that step | Targeted detail is cheaper than logging everything |
Worked example
Situation: A support platform runs a router, a retrieval step and two specialist agents. Customer satisfaction is falling, yet error rates and average latency look normal.
Decision: Add tracing with one request ID across all steps, record tokens, cost and p95 latency per step, and score a daily sample of conversations with an LLM judge calibrated on human ratings.
Why: Nothing is erroring, so system metrics alone won't show the cause; sampled evals reveal the quality drop and traces show which step is responsible. More verbose per-service logs would still leave engineers stitching steps together by hand.
Practice
A travel company's booking assistant uses a planner agent, three sub-agents and eight tools. About 3% of itineraries have wrong dates. Each service writes its own plain-text logs, so engineers cannot tell which step introduced an error. What should the architect put in place first?
- Raise every service's log level to debug and collect the files centrally.
- Add a nightly LLM judge that scores a sample of finished itineraries.
- Add a feedback button so customers can flag wrong itineraries.
- Trace each request end to end under one ID across agents and tools.
Show answer
Answer: D. The deciding fact: engineers cannot tell which step failed. A shared request ID with a span per step lets them open a bad itinerary and see where the date went wrong.
- A ❌ Works for detail, but without a shared ID the steps still can't be joined into one story.
- B ❌ Works to measure how often dates are wrong, but not to show which step caused it.
- C ❌ Works to find bad cases, but still gives no view of the steps behind them.
- D ✅ Trace each request end to end under one ID across agents and tools.
A legal-research product's monthly model spend doubled while request volume stayed flat. Average latency looks normal, but some users report very slow answers. The current dashboard shows only total daily tokens and mean latency. Which monitoring change best supports the diagnosis?
- A budget alert that fires when monthly token spend passes a limit.
- Tokens, cost and latency per request and step, with p95 and p99.
- A weekly LLM-as-judge evaluation on a random sample of answers.
- Full prompt and response text stored for every request.
Show answer
Answer: B. The deciding facts: totals and averages hide where the extra spend and the slow tail come from. Per-request, per-step figures with percentiles point to the expensive or slow path.
- A ❌ Works to warn about spend, but not to explain where it goes.
- B ✅ Tokens, cost and latency per request and step, with p95 and p99.
- C ❌ Works for quality drift, which is not the symptom here.
- D ❌ Works for deep debugging of one case, but it is costly at scale and gives no aggregate view.
3.5 · Design a RAG pipeline with appropriate chunking and indexing strategies
Guide line: “Design a RAG pipeline with appropriate chunking and indexing strategies”
RAG (retrieval-augmented generation) finds relevant passages and gives them to the model with the question. Before retrieval can work, documents are split into chunks, each turned into an embedding (a list of numbers that captures meaning) and stored in an index. How you chunk decides what can be found: a chunk that cuts a clause in half can never be retrieved whole.
Fixed-size chunking (every 500 characters) is simple but ignores meaning. Structure-aware chunking splits on the document's own boundaries: headings, sections, clauses, list items, table rows kept together. A little overlap between neighbouring chunks protects ideas that cross a boundary. Chunk size is a trade-off: small chunks match precisely but lack context; large chunks carry context but dilute the match. A common pattern retrieves small chunks and then passes their larger parent section to the model.
Chunks often lose the context that made them meaningful (“the limit is 25 days” without saying which country). Two fixes. Attach metadata (source, section, date, version, region, access list) for filtering and citation. Use contextual retrieval, a technique Anthropic has described: before indexing, prepend each chunk with a short, model-written note situating it in its document, and index that for both embedding and keyword search. Finally, re-index when documents change: update changed files, delete removed ones, and embed queries with the same embedding model as the index.
Know this
- Structure-aware chunking (by section, clause or heading) beats fixed-size chunking for structured documents.
- Overlap between chunks protects ideas that cross a boundary.
- Small chunks match precisely but lack context; large chunks carry context but match loosely.
- Metadata such as source, date, version, region and access list enables filtering and citations.
- Contextual retrieval prepends a short document-aware summary to each chunk before embedding and keyword indexing.
- Re-index when sources change, remove deleted documents, and use the same embedding model for queries and index.
- Tables should stay whole, or be stored as structured data, rather than split mid-row.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| A clause or table is split across two chunks | Structure-aware chunking with overlap | Keeps each idea in one retrievable piece |
| Chunks are correct text but from the wrong version, region or date | Metadata plus filters, or contextual retrieval | The chunk alone doesn't say where it belongs |
| Short, unstructured documents such as chat messages or notes | Simple fixed-size or whole-document chunks | Little structure to exploit; simplicity wins |
| Answers became wrong right after a document refresh | Check the re-index and embedding model | Stale or mismatched index, not the model |
| Matches are precise but answers lack surrounding detail | Retrieve small chunks, pass the parent section | Precision for search, context for the answer |
Worked example
Situation: A manufacturer's maintenance assistant indexes equipment manuals in fixed 1,000-character chunks. Technicians get half a procedure: step 4 onwards sits in a different chunk from steps 1 to 3, and warnings tables are split mid-row.
Decision: Re-chunk by heading and procedure, keep each procedure and each table whole, add modest overlap, and tag chunks with machine model and manual version.
Why: The failure is in how text was cut, so the fix is in chunking. Raising top-k might fetch both halves by luck but also floods the context with unrelated text; a bigger model cannot reason over steps it never received.
Practice
An insurer's policy assistant splits PDFs into fixed 500-character chunks. Answers often miss exclusions because a clause's condition and its exception land in different chunks. The policies use numbered sections and clauses throughout. What should the architect change?
- Raise top-k from 5 to 20 so both halves of a clause are more likely retrieved.
- Switch to a larger embedding model with more dimensions.
- Chunk by section and clause, with some overlap and section metadata.
- Shrink chunks to 200 characters so each one matches more precisely.
Show answer
Answer: C. The deciding facts: ideas are split by fixed cuts, and the documents have clear numbered structure. Chunking along that structure keeps each clause and its exception together.
- A ❌ Works sometimes, but by luck, and it adds unrelated text; the split remains.
- B ❌ Works for better meaning-matching, but it cannot join text that chunking separated.
- C ✅ Chunk by section and clause, with some overlap and section metadata.
- D ❌ Works for precision on short facts, but here it splits clauses even more.
A global firm's HR knowledge base holds hundreds of near-identical policy documents, one per country and year. Retrieved chunks such as “Employees receive 25 days” are accurate text but often from the wrong country or an outdated year, and the chunks themselves name neither. Which two changes best fix retrieval?
- Prepend each chunk with a short note naming its document, country and year.
- Increase the overlap between neighbouring chunks.
- Switch from semantic search to keyword-only search.
- Store country and effective year as metadata and filter on them.
- Raise top-k so more candidate chunks reach the model.
Show answer
Answer: A and D. The deciding fact: the chunks name neither country nor year. Contextual notes put that context into what gets indexed, and metadata filters restrict results to the right country and current year.
- A ✅ Prepend each chunk with a short note naming its document, country and year.
- B ❌ Works when ideas cross chunk boundaries, but the country and year are not in nearby text either.
- C ❌ Works for exact codes and names, but the chunks still don't contain the country or year to match.
- D ✅ Store country and effective year as metadata and filter on them.
- E ❌ Works when the right chunk ranks just below the cut-off, but here more results bring more wrong-country text.
3.6 · Apply retrieval strategies matched to data shape and query pattern
Guide line: “Apply retrieval strategies matched to data shape and query pattern”
Semantic (vector) search compares embeddings, so it finds passages with similar meaning even when the words differ. It is weak on exact strings: product codes, error codes, part numbers and names can look alike in meaning space. Keyword search, usually with the BM25 ranking method, scores documents by matching the exact terms, so it excels at codes and names but misses paraphrases.
Hybrid search runs both and merges the results, which suits collections where users mix exact identifiers with descriptions. Metadata filters (date, region, product version, owner) narrow the search before or during ranking. A reranker takes the top few dozen candidates and re-scores them with a more precise model, lifting the right passage from position 30 into the top 5 at some cost in latency.
Some data should not be embedded at all. Questions about totals, counts, trends or filters over tables (“revenue by region last quarter”) need exact computation over every row, so give the model a query tool over the database or warehouse, ideally one with defined metrics, and let it answer from the result. Retrieving the 20 most “similar” rows cannot add up two million.
Know this
- Vector search matches meaning; it is weak on exact identifiers such as codes, SKUs and names.
- BM25 keyword search matches exact terms; it misses paraphrases and synonyms.
- Hybrid search combines both for collections where users mix codes with descriptions.
- Metadata filters handle date, region, version and owner constraints precisely.
- A reranker improves precision when the right result is retrieved but ranked too low.
- Aggregations and exact figures over structured data need a query tool, not embeddings.
- A better embedding model still matches by meaning, so it does not fix exact-match failures.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| Users describe problems in their own words | Vector (semantic) search | Matches meaning across different wording |
| Queries contain exact codes, IDs or names, mixed with descriptions | Hybrid: vector plus BM25 keyword | Exact terms and meaning both count |
| Queries contain only exact identifiers, such as a part-number lookup | Keyword search or a direct lookup | No meaning to match; exact is enough |
| The right passage is retrieved but ranked too low | Add a reranker | Re-scores candidates for precision |
| Questions need totals, counts or trends over tables | A query tool over the database | Exact computation over every row |
Worked example
Situation: A retailer's merchandising assistant answers both “What do customers say about the waterproof jackets?” and “How many units of SKU 88213 sold in Germany in March?” from one vector index of reviews and sales rows.
Decision: Keep semantic search over reviews; route sales questions to a tool that queries the sales warehouse with filters for SKU, country and month.
Why: Reviews are text where meaning matters; sales are structured numbers that need exact aggregation. Hybrid search would help find the SKU in text, but no search method can sum the units correctly.
Practice
A sales team asks an assistant questions such as “What was total revenue by region last quarter?” The data sits in a warehouse table of two million order rows. The current design embeds each row as text and retrieves the 20 most similar rows. Answers sound plausible but the numbers are wrong. What should the architect use?
- A tool that queries the warehouse and returns the aggregated figures.
- Hybrid search combining vector and BM25 keyword matching over rows.
- A reranker that re-scores the top 100 retrieved rows.
- Metadata filters on region and quarter, applied before the vector search runs.
Show answer
Answer: A. The deciding fact: the questions need totals over millions of rows. Only an exact query can compute them; any retrieval method hands the model a small sample.
- A ✅ A tool that queries the warehouse and returns the aggregated figures.
- B ❌ Works for exact terms in text, but still returns a handful of rows, not a total.
- C ❌ Works to improve ranking, but 100 rows still cannot sum two million.
- D ❌ Works to narrow the rows, but the model still sees only a sample of them.
A software company's support search covers 50,000 troubleshooting articles. Customers paste exact error codes such as E-4021 and also describe symptoms in their own words. Vector search alone often returns articles about similar-looking codes, and the right article is usually in the top 50 but rarely in the top 5. Which two changes best fit?
- Fine-tune a new embedding model on the article collection.
- Add BM25 keyword search and merge its results with vector results.
- Replace vector search with keyword search alone.
- Filter articles by product-version metadata before searching.
- Rerank the top 50 candidates and pass the best few to Claude.
Show answer
Answer: B and E. Two deciding facts: queries mix exact codes with descriptions, which calls for hybrid search; and the right article is retrieved but ranked too low, which a reranker fixes.
- A ❌ Works to sharpen meaning-matching, but embeddings still blur exact codes.
- B ✅ Add BM25 keyword search and merge its results with vector results.
- C ❌ Works for codes, but loses the symptom descriptions customers also write.
- D ❌ Works when results come from the wrong version, which the stem doesn't mention.
- E ✅ Rerank the top 50 candidates and pass the best few to Claude.
3.7 · Evaluate connection protocols and select the appropriate integration mechanism (MCP, API/CLI, agent-to-agent)
Guide line: “Evaluate connection protocols and select the appropriate integration mechanism (MCP, API/CLI, agent-to-agent)”
The Model Context Protocol (MCP) is an open standard for exposing tools, data (resources) and reusable prompts to AI applications. A team builds one MCP server, and any MCP-capable client can use it: Claude Code, the Claude desktop app, agents built with the Claude Agent SDK, or the Claude API through its MCP connector. Servers can run locally or remotely. MCP pays off when the same capability is needed by several clients or teams, because it replaces one-off glue code with one shared, governed integration.
A direct API call is the lightest option: your application code calls a known service, or defines a tool in its own code that wraps that call. It fits one application with a fixed integration. A CLI (command-line tool) fits agents that have a shell, such as Claude Code: well-known tools like git or a cloud CLI are already familiar to the model and need no extra server. Building an MCP server for a single, fixed call inside one app adds moving parts for no reuse.
Agent-to-agent integration means delegating a goal, not a function call, to another autonomous agent that plans its own steps with its own tools and data, often owned by another team or company. Open protocols such as A2A define how agents describe themselves, accept tasks and return results. Choose it when the other side keeps its logic and systems private and runs a multi-step process itself. The general rule: pick the lightest mechanism that meets the reuse and ownership needs.
Know this
- MCP standardises how tools, resources and prompts are exposed, so one server serves many AI clients.
- MCP pays off when several clients or teams need the same capability.
- A direct API call is simplest for one application with a fixed, known integration.
- Agents with a shell can use familiar CLIs directly, without building a server.
- Agent-to-agent delegates a goal to an autonomous agent that runs its own multi-step process.
- Agent-to-agent fits when the other party owns the logic and won't expose its underlying systems.
- Pick the lightest mechanism that satisfies reuse, ownership and security needs.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| The same tool is needed by several AI clients or teams | An MCP server | Build once, reuse everywhere, govern centrally |
| One backend calls a known service in a fixed way | Direct API call | Fewest moving parts; nothing to reuse |
| A coding agent with a shell needs a well-known tool | The existing CLI | The model already knows it; no server needed |
| A partner's agent handles a multi-step task with its own private systems | Agent-to-agent | You hand over a goal, not individual calls |
| A partner offers simple, documented endpoints and no agent | Direct API or an MCP wrapper | There is no agent to delegate to |
Worked example
Situation: A bank's ticketing data must be searchable from Claude Code by developers, from the desktop app by support leads, and by two internal agents. Separately, a nightly script sends each day's new tickets to Claude once for a summary.
Decision: Build one MCP server for ticket search and read, used by all four clients. Keep the nightly summary as a direct API call in the script.
Why: Four clients share one capability, so MCP removes duplicated glue. The nightly job is a single fixed call; wrapping it in MCP would add a server to run for no benefit.
Practice
Four teams want Claude to read the company's incident database: developers in Claude Code, analysts in the Claude desktop app, and two internal agents built on the Agent SDK. Each team has started writing its own integration code, and security wants one place to control access. Which mechanism fits?
- A shared REST API that each client team wraps in its own tool definitions.
- A command-line tool installed on every user's machine.
- An incident agent that the other agents message with questions.
- One MCP server exposing incident search and read to every client.
Show answer
Answer: D. The deciding fact: four different AI clients need the same capability. One MCP server serves them all and gives security one place to control access.
- A ❌ Works, but each team still writes and maintains its own wrapper; it fits when only one app needs it.
- B ❌ Works for developers with a shell, but not for the desktop app or hosted agents.
- C ❌ Works when incidents need multi-step investigation by their own agent, but here clients just read data.
- D ✅ One MCP server exposing incident search and read to every client.
A travel company's assistant must rebook hotel stays with a partner chain. The partner runs its own booking agent that checks availability, applies loyalty rules and takes payment over several steps, and will not expose its internal systems. Which integration mechanism fits?
- Call the partner's public availability API and apply loyalty rules ourselves.
- Wrap the partner's public website in a browser-automation tool.
- Hand the rebooking task to the partner's agent over an agent-to-agent protocol.
- Build an MCP server that exposes each partner booking step as a tool.
Show answer
Answer: C. The deciding facts: the partner runs the multi-step process itself and keeps its systems private. Delegating the goal to its agent is the only option that respects both.
- A ❌ Works for availability, but it duplicates loyalty and payment logic the partner owns and won't share.
- B ❌ Works as a last resort, but it is brittle and bypasses the partner's intended interface.
- C ✅ Hand the rebooking task to the partner's agent over an agent-to-agent protocol.
- D ❌ Works when the partner exposes its booking steps, which it refuses to do here.
3.8 · Evaluate progressive discovery vs. monolithic context strategy
Guide line: “Evaluate progressive discovery vs. monolithic context strategy”
A monolithic context strategy loads everything into every request: all instructions, all reference documents, all tool definitions. It is simple, predictable, and works well with prompt caching. But as it grows it costs more tokens, slows responses, and dilutes attention: with hundreds of similar tool definitions, the model is more likely to pick the wrong one.
Progressive discovery (also called progressive disclosure) starts with a short catalogue and loads detail only when needed. Claude's mechanisms include Skills, where only each Skill's name and description sit in context until a task calls for it, then its instructions and any extra files are read; tool search, where tool definitions are deferred and Claude searches for the few it needs; MCP resources, data a server offers to be read on demand; and simply letting an agent browse files.
Neither is always right. Progressive discovery adds a lookup step and a small chance of missing something relevant. Monolithic wins when the content is small, stable and needed on almost every request; then caching makes it cheap. Progressive discovery wins when the content is large, varied, and each task uses only a slice.
Know this
- Monolithic context is simple and cache-friendly, but grows costly and dilutes attention as content grows.
- Progressive discovery keeps a short index in context and loads detail on demand.
- Skills load only their name and description until needed, then their full instructions and files.
- Tool search lets Claude find relevant tools from a large set instead of loading every definition.
- MCP resources expose data a client can read when needed rather than preloading it.
- Monolithic wins when content is small, stable and needed every time; prompt caching then makes it cheap.
- A bigger context window does not fix tool confusion or attention dilution.
Mind map
| When the stem says… | Pick | Why |
|---|---|---|
| Hundreds of tools, each task uses a few, and wrong tools get picked | Tool search (deferred tool loading) | Only relevant definitions reach the model |
| Long procedures, each needed only for certain tasks | Skills | Instructions load only when the task calls for them |
| Small, stable content needed on nearly every request | Monolithic, with prompt caching | No lookup step; caching makes it cheap |
| Large reference data read occasionally | MCP resources or file browsing | Fetched on demand instead of preloaded |
Worked example
Situation: A platform-engineering agent carries 150 tool definitions from five MCP servers and a 30-page runbook in every request. Typical tasks use three tools and one runbook section; replies are slow and it sometimes calls a similar but wrong tool.
Decision: Defer tool definitions behind tool search, and split the runbook into Skills by procedure, each with a clear description.
Why: Each task uses a small slice of a large, varied set, which is where progressive discovery wins. Caching the full context would cut cost and time to first token but would leave all 150 tools competing for the model's attention.
Practice
A company's operations agent loads 180 tool definitions from six MCP servers and a 40-page runbook into every request. Each task uses only three to five tools and one runbook procedure. Responses are slow, costs are high, and the agent often picks a similar but wrong tool. What should the architect change?
- Move to a model with a larger context window.
- Load tools through tool search and split the runbook into Skills.
- Cache the full tool list and runbook with prompt caching.
- Merge the 180 tools into a dozen broad multi-purpose tools.
Show answer
Answer: B. The deciding fact: each task uses a small slice of a large set, and the agent confuses similar tools. Loading only what the task needs fixes cost, speed and tool choice together.
- A ❌ Works to fit more, but more room does not reduce cost or tool confusion.
- B ✅ Load tools through tool search and split the runbook into Skills.
- C ❌ Works to cut cost and time to first token, but all 180 tools still compete for attention.
- D ❌ Works to shorten the list, but broad tools are vague and carry more power than each task needs.
A support bot for a single product uses a 3,000-token style guide and six tools in every conversation. Every reply needs all of them, and the content changes about once a quarter. A colleague proposes converting the guide into Skills and the tools to tool search to keep context lean. What should the architect recommend?
- Adopt Skills and tool search so the starting context stays small.
- Expose the style guide as MCP resources the bot reads when needed.
- Index the style guide for retrieval and fetch sections per message.
- Keep everything loaded up front and cache the stable prefix.
Show answer
Answer: D. The deciding facts: the content is small, stable, and needed on every reply. Loading it monolithically with prompt caching is cheap and avoids a lookup step that would run every time anyway.
- A ❌ Works when content is large and only sometimes needed; here it would load on every reply anyway.
- B ❌ Works for large reference data read occasionally, not a small guide used every time.
- C ❌ Works for big document sets, but risks missing style rules that every reply must follow.
- D ✅ Keep everything loaded up front and cache the stable prefix.
Source of the guide lines: Claude Certified Architect – Professional Exam Guide, v1.0 (Anthropic). Explanations and questions are ours, written fresh. · CCA-P start here