CONCEPT Prompt engineering is empirical, not vibes
Good prompting is an empirical discipline, not a matter of taste. You don't "know" a prompt is better — you measure it. Anthropic's guidance assumes you arrive with three prerequisites before you ever tune a word:
| Prerequisite | What it means | Why it must come first |
|---|---|---|
| 1. Success criteria | A clear, specific definition of what "good output" is | You can't improve toward an undefined target |
| 2. Empirical test | A way to score outputs against those criteria (an eval) | Turns "feels better" into a number you can compare |
| 3. First-draft prompt | Something concrete to improve | The loop refines a draft; it doesn't invent from nothing |
CONCEPT The loop: define → test → refine → repeat
With the prerequisites in place, prompt engineering becomes a tight feedback loop. You define success, build a set of test cases that cover real and edge inputs, run the current prompt, score against the criteria, then refine and run again. Each turn of the loop is cheap (seconds), which is exactly why prompting is the lever you iterate on.
Make your success criteria specific and measurable. "Good summary" is unscorable; "≤ 3 sentences, names every party, no invented facts" is a rubric an eval can check. And build a diverse test set — cover the common case and the edge cases, so a prompt that helps one input doesn't silently break another.
DIAGRAM The define → test → refine cycle
(specific, measurable)"] --> B["Build test cases
(common + edge)"] B --> C["Draft / edit the prompt"] C --> D["Run prompt on test set"] D --> E{"Meets criteria
across cases?"} E -->|"no"| F["🟠 Refine prompt
one change at a time"] F --> D E -->|"yes"| G["🟢 Ship it
keep the eval as a regression net"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef start fill:#cfe2ff,stroke:#0d6efd,color:#084298; class G ok; class F gate; class A start;
CODE A minimal eval harness — the loop in code
The refine step is just editing PROMPT; everything else stays fixed. Note how the criteria are
encoded as checks, not judged by eye — that's what makes each iteration comparable.
# 1. SUCCESS CRITERIA, encoded as scorable checks. def score(output, case): ok = len(sent_split(output)) <= 3 # ≤ 3 sentences ok &= all(p in output for p in case["must_name"]) # names all parties ok &= not has_hallucination(output, case["source"]) # no invented facts return ok # 2. TEST CASES: common + edge (short doc, missing party, adversarial). CASES = load_cases("summary_eval.jsonl") # 3. The only thing you edit between iterations: PROMPT = "Summarize in ≤3 sentences. Name every party. Use only the source text." # 4. RUN + SCORE the whole set — one number to compare against last run. passed = sum(score(run(PROMPT, c), c) for c in CASES) print(f"{passed}/{len(CASES)} passed") # refine PROMPT, re-run, compare
TRADE-OFFS Empirical loop vs. eyeballing one output
✅ Test-driven loop
- Improvements are measured, not felt
- Regressions on edge cases caught instantly
- The eval doubles as a permanent regression net
- Changes are comparable run-to-run
⚠️ Eyeballing a single example
- Fast to start, but you optimize one input
- Silent regressions on cases you didn't look at
- "Better" is subjective and non-reproducible
- No safety net when you change the prompt later
EXAM TRAPS
PRACTICE
Q1 · What are the three prerequisites before prompt engineering?
(1) A clear, specific definition of success criteria; (2) a way to empirically test outputs against them (an eval); and (3) a first-draft prompt to improve. The loop refines a draft against a measurable target — all three must exist first.
Q2 · Why build a diverse test set instead of one example?
Because a prompt change that helps one input can silently break another. A diverse set covering common and edge cases turns each iteration into a full regression check, so improvements are real and regressions surface immediately.
Q3 · What makes success criteria usable in the loop?
They must be specific and measurable — encodable as checks (e.g. "≤ 3 sentences, names all parties, no invented facts") rather than subjective ("good"). Only measurable criteria let you score runs and compare iterations objectively.