Extraction honesty
When a document doesn't say something, a badly designed extraction makes it up. The trap answers push harder for a value: make the field required, retry until it passes, pick the nearest category. The exam's answers give the model an honest way out: null, “other” plus detail, a specific error when the value is on the page, and a person when it isn't.
D4 Prompt Engineering & Structured OutputExam guide 4.3, 4.4 and 4.2About 15 minutes
- Schema and nullable fields: A schema stops syntax errors but not wrong values, so make a field optional or nullable when the value may not be in the document, and retry with the specific error only when the information is actually there.
- Retry with the error: When an extraction fails a check, retry with the specific error attached, but don't retry when the value isn't in the document at all.
Start here: the temp and the form screen
An office temp copies paper forms into a computer system. The screen won't save until every box is filled.
Two more habits help. When the supervisor sends a form back, “you swapped the start and end dates” gets it fixed in one go; “wrong, redo it” gets the same mistake again. And if the date is torn off the paper, no amount of “try again” brings it back: that form goes to someone who can phone the customer. Showing the temp three finished examples of the odd layouts helps them get it right first time.
Claude is the temp. The JSON schema on the tool is the form screen. Asking for JSON in the reply is a screen with no boxes at all: the syntax can break. A required field with nothing on the page is the box that must be filled.
- It isn't there: let the schema say so. An optional or nullable field, and a person if the business needs the value. Never a required field, a sentinel like 0, or another retry.
- It's there, but doesn't fit the list: an enum with “other” plus a detail field (or “unclear”).
- It's there, but extracted wrong: retry with the specific validation error. For layouts that trip it up every time, add worked examples.
- The JSON itself breaks: return it through a tool with a JSON schema, not as text in the reply.
Words in the question that tell you which side you're on
Not on the page
- “many documents don't show …”, “cut off”, “left blank”
- “returns 0”, “copies another field”, “looks real”
- “a new category”, “forced into the nearest value”
- “the retries keep failing” on a missing value
On the page, extracted wrong
- “both dates are printed clearly”
- “the retry message says only ‘try again’”
- “accuracy drops on carriers' own layouts”
- “adds a sentence after the JSON”, “a missing bracket”
Try it: run the certificates
Pick a certificate, then set up the design. The simulator shows what Claude returns, what the validator says, each retry, and what finally gets posted. The strip at the bottom runs all five certificates through the same design.
The behaviours are illustrative, chosen to show the shape. They are not measurements, and every run is the same. The exam guide's points are that optional or nullable fields prevent fabrication, that a schema removes syntax errors but not semantic ones, and that retries help only when the information is in the document.
Things to try: No deductible printed with Required, then Sentinel, then Optional / nullable. Then Policy number cut off with Retry with the specific error: the retry produces an invented number. Then Carrier's own schedule with “try again”, then the specific error, then worked examples. Last, set every row to its last choice and read the strip.
The three traps
| Trap | What the wrong option looks like | Why it loses |
|---|---|---|
| Forcing a value | Keep the field required; use 0 or “N/A” for missing; map new categories to the nearest listed value | The model fills the box with something plausible. Nothing downstream can tell an invented value from a printed one. |
| Retrying the impossible | More retries, a firmer retry message, or examples, when the value isn't in the document | Retries fix values that are on the page. Pressing for one that isn't invites an invention. Send it to a person. |
| Fixing the wrong layer | A blind “try again” for a mapping slip; a parser that cleans up JSON text; a prompt rule “do not guess” | The specific error tells the model what to fix. A tool with a schema removes syntax errors. A schema with a null option beats a rule asking for honesty. |
Practice: three exam-style questions
Answer first, then open “Think it through”.
Rule to remember
Exam guide 4.3: “Optional/nullable fields prevent fabrication; enum "unclear"/"other" + detail for extensible categories.” and “Strict schemas eliminate syntax errors but not semantic errors (line items not summing, wrong fields).” Exam guide 4.4: “Retries are ineffective when info is absent from the source (vs format/structural errors).”