Agree it in numbers
D6 · Stakeholders & LifecycleGuide lines 6.3, 1.6
Sponsors often approve an AI project with words like “accurate”, “quick” or “never wrong”. Those words cannot be checked, so months later the sponsor and the team argue about whether the system is good enough, and nobody can settle it. The architect's job is to turn the words into measurable targets: an accuracy level on sampled real traffic, a reply time at a percentile (for example, 95% of replies within three seconds), and a cost per outcome.
Two habits make those targets fair. First, measure today's process on the same cases: people make mistakes too, and the honest question is whether the new system beats the current one. Second, every error target comes with an error-handling plan: who reviews, how mistakes are corrected and when cases are escalated.
The target also follows the value pillar the sponsor cares about (cost, productivity, speed, accuracy or a new capability). Once targets are agreed, pick the design that meets them for that pillar. The exception: when a single error could cause serious harm, a target is not enough, and every output is checked.
Change one fact
A water utility's sponsor will approve an assistant that drafts and sends replies to billing complaints, but only if it is “accurate and quick”. On 400 past complaints its replies were correct 92% of the time; agents writing from scratch were correct 89% of the time. The team says that is good enough; the sponsor is unsure. What should the architect do?
Words in the stem that signal this trap
- “must be 100% accurate” or “never wrong”
- “fast and accurate”, with no numbers
- “nobody has measured how the current team does”
- “the sponsor's priority is…” (cost, productivity, speed)
- “an average reply time” against a promised percentile
- “a wrong answer could cause serious harm”
Practice
A logistics firm's operations head says a customs-code assistant must beat the clerks before it launches. On 600 past shipments it picks the right code 93% of the time. Nobody has measured how often the clerks pick the right code; the head believes they are “near perfect”. What should the architect do next?
- Tune the prompt until the assistant reaches 99%, which must be above the clerks.
- Launch with every code checked by a clerk, so the comparison no longer matters.
- Present the 93% result with examples of the hardest codes it got right.
- Measure the clerks on the same 600 shipments and agree a target from both.
Show answer
Answer: D. The deciding fact: the clerks' accuracy is a belief, not a measurement. Measuring them on the same cases gives the fair baseline the target needs.
- A ❌ Works to raise accuracy, but 99% is a guess at an unmeasured bar, and tuning may never reach it.
- B ❌ Works when errors are severe, but it dodges the agreed question and gives up the time saving.
- C ❌ Works to show strengths, but examples do not answer whether it beats the clerks.
- D ✅ Measure the clerks on the same 600 shipments and agree a target from both.
A parcel company's contract with a partner says its tracking assistant must reply within three seconds for 95 percent of requests. The engineering dashboard shows an average reply time of 1.8 seconds, so the team reports the target as met. The partner says many replies take far longer. What should the architect do first?
- Switch the live path to a smaller, faster model to bring every reply time down.
- Report the 95th-percentile reply time from the same logs against the contract.
- Turn on streaming so customers see the first words of each reply sooner.
- Share the monthly average with the partner, backed by a month of daily figures.
Show answer
Answer: B. The deciding fact: the contract names a 95th percentile, and an average can hide a slow tail. Measuring the figure the contract uses settles who is right before anything changes.
- A ❌ Works if the tail is too slow, but it changes the system before anyone knows the real number.
- B ✅ Report the 95th-percentile reply time from the same logs against the contract.
- C ❌ Works for how fast replies feel, but it does not change the measured reply time.
- D ❌ Works for showing a trend, but an average is not the figure the contract promises.
An immigration law firm's assistant fills in visa forms from client documents. On 300 past cases, 97% of forms were fully correct. One wrong field gets an application refused, losing the fee and months of the client's time, and it cannot be fixed after filing. Partners ask what quality plan to agree. What should the architect propose?
- A 97% accuracy target, tracked on a monthly sample of the forms filed.
- Measure the paralegals' own error rate first, then set a target above it.
- A paralegal check of every form before filing, with errors added to the eval.
- More prompt tuning until all 300 test cases come out with no errors.
Show answer
Answer: C. The deciding fact: one error causes serious harm that cannot be undone after filing. Error handling must cover every form, so each is checked before it goes.
- A ❌ Works when errors can be caught and fixed later; here a sampled error is already filed and refused.
- B ❌ Works when errors are recoverable, but any accepted error rate still lets refusals through.
- C ✅ A paralegal check of every form before filing, with errors added to the eval.
- D ❌ Works to raise the score, but a clean test set does not make filed forms error-free.