CCA-F start here · Trap gym · Confidence and sampling

Confidence and sampling

An extraction pipeline scores 97% overall, and the team wants to switch off human review. The tempting answers trust that number, or a thin random sample, or Claude's own sense of how sure it is. Each can hide one document type or field that fails badly. The exam's answer is to measure by type and field with stratified random samples, and route review by calibrated field confidence.

D5 Context & ReliabilityExam guide 5.5About 12 minutes

At a glance: a high overall accuracy can hide one weak document type or field, so measure accuracy by type and field with stratified random samples before automating.

Start here: the school average

A school's average exam score is 85%, so the head says every class is doing fine.

The average. One number for the whole school. A class of 30 scoring 40% barely moves it, because hundreds of other pupils are doing well.
Class by class. Someone checks each class, and the one at 40% stands out at once. Only looking at each group showed it.

An extraction pipeline is the same. Claude reads invoices, receipts, scans and handwritten forms and pulls out fields such as the amount. Stratified random sampling means picking a random sample from each group (each document type), so a rare type gets enough documents to measure. Calibrated confidence means a confidence score per field that has been checked against a labelled set (documents where people wrote down the right answers), so “90% confident” really means right about 90% of the time. Then low-scoring fields go to a person.

Documents four types Claude extracts four fields each Route each field Posts by itself A person checks the check decides which fields may post Overall accuracy Random sample Claude rates itself Stratified sample by type Calibrated field confidence one number rare types barely appear never checked each type and field measured on a labelled set, routes each field
The one question to ask: has accuracy been checked for each document type and field, or only as one overall number?

Words in the question that tell you which side you're on

Check by type and field

  • “97% overall”, “accuracy across all documents”
  • “ready to automate”
  • “one vendor's invoices”, “handwritten forms”
  • “a random sample”, “only a handful were in the set”

Calibrated field-level routing

  • “which fields to review”
  • “high-confidence errors”
  • “labelled set”, “calibrate”
  • “the model returns a confidence for each field”

Try it: check before you automate

The pipeline reads 10,000 documents a month, four fields each. Handwritten forms are only 2% of them. Pick how you check before switching off review. Every method gets the same budget: 400 labelled documents. The grid shows what your check lets you see; a field posts by itself only if it measures at least 97%.

How you check before automating

Wrong values posted without review, each month

Share of fields checked by people

The accuracies, volumes and catch rates are illustrative, chosen to show the shape. The samples are drawn with a fixed seed, so each choice always gives the same grid. The exam guide's point is that aggregate accuracy can mask weak document types and fields.

Things to try: overall accuracy only, then press “Show what is really true” and find the handwritten row. Then a plain random sample: count the handwritten forms it picked. Then stratified by type, and last stratified + calibrated: watch the pink meter shrink.

The four traps

TrapWhat the wrong option looks likeWhy it loses
Masks weak spots“Automate fully because overall accuracy is 97%”It works for most documents, but the average can hide a type or field that fails badly.
Too thin“Have people review a random 5% of all documents”A useful spot check, but an unstratified sample can hold almost none of the rare, weak document type.
Uncalibrated“Send only low-confidence extractions to human review”It catches some errors, but without calibration and sampling, confident errors pass straight through unseen.
Unreliable proxy“Ask Claude to rate its own confidence on every field”A signal, but self-rated confidence has to be calibrated against labelled data before it can route review.

Practice: three exam-style questions

Answer first, then open “Think it through”.

Rule to remember

Don't trust the average: check each type and field before you automate. Sample each type on its own, and route review by field confidence calibrated on labelled data.

Exam guide 5.5: “Aggregate accuracy (97%) can mask poor performance on specific doc types/fields.” “Stratified random sampling measures high-confidence error rates; field-level confidence calibrated on labeled sets routes review.” “Validate accuracy by document type and field before automating.”

← All traps