D4 · Prompt Engineering › Cluster 1 · Foundations & Being Clear

1.2 · The Empirical, Iterative Loop

Define success · test · refine

CONCEPT Prompt engineering is empirical, not vibes

Good prompting is an empirical discipline, not a matter of taste. You don't "know" a prompt is better — you measure it. Anthropic's guidance assumes you arrive with three prerequisites before you ever tune a word:

PrerequisiteWhat it meansWhy it must come first
1. Success criteriaA clear, specific definition of what "good output" isYou can't improve toward an undefined target
2. Empirical testA way to score outputs against those criteria (an eval)Turns "feels better" into a number you can compare
3. First-draft promptSomething concrete to improveThe loop refines a draft; it doesn't invent from nothing
If you're missing any of these, stop and establish them first. Tuning a prompt with no eval is guessing — you'll chase one example and regress on ten others you never checked.

CONCEPT The loop: define → test → refine → repeat

With the prerequisites in place, prompt engineering becomes a tight feedback loop. You define success, build a set of test cases that cover real and edge inputs, run the current prompt, score against the criteria, then refine and run again. Each turn of the loop is cheap (seconds), which is exactly why prompting is the lever you iterate on.

Make your success criteria specific and measurable. "Good summary" is unscorable; "≤ 3 sentences, names every party, no invented facts" is a rubric an eval can check. And build a diverse test set — cover the common case and the edge cases, so a prompt that helps one input doesn't silently break another.

The empirical loop is what makes prompt engineering trustworthy: every change is validated against the same test set, so improvements are real and regressions are caught immediately.

DIAGRAM The define → test → refine cycle

flowchart TD A["Define success criteria
(specific, measurable)"] --> B["Build test cases
(common + edge)"] B --> C["Draft / edit the prompt"] C --> D["Run prompt on test set"] D --> E{"Meets criteria
across cases?"} E -->|"no"| F["🟠 Refine prompt
one change at a time"] F --> D E -->|"yes"| G["🟢 Ship it
keep the eval as a regression net"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef start fill:#cfe2ff,stroke:#0d6efd,color:#084298; class G ok; class F gate; class A start;

CODE A minimal eval harness — the loop in code

The refine step is just editing PROMPT; everything else stays fixed. Note how the criteria are encoded as checks, not judged by eye — that's what makes each iteration comparable.

# 1. SUCCESS CRITERIA, encoded as scorable checks.
def score(output, case):
    ok  = len(sent_split(output)) <= 3          # ≤ 3 sentences
    ok &= all(p in output for p in case["must_name"])  # names all parties
    ok &= not has_hallucination(output, case["source"])   # no invented facts
    return ok

# 2. TEST CASES: common + edge (short doc, missing party, adversarial).
CASES = load_cases("summary_eval.jsonl")

# 3. The only thing you edit between iterations:
PROMPT = "Summarize in ≤3 sentences. Name every party. Use only the source text."

# 4. RUN + SCORE the whole set — one number to compare against last run.
passed = sum(score(run(PROMPT, c), c) for c in CASES)
print(f"{passed}/{len(CASES)} passed")   # refine PROMPT, re-run, compare

TRADE-OFFS Empirical loop vs. eyeballing one output

✅ Test-driven loop

  • Improvements are measured, not felt
  • Regressions on edge cases caught instantly
  • The eval doubles as a permanent regression net
  • Changes are comparable run-to-run

⚠️ Eyeballing a single example

  • Fast to start, but you optimize one input
  • Silent regressions on cases you didn't look at
  • "Better" is subjective and non-reproducible
  • No safety net when you change the prompt later
🔑 Upfront cost of building an eval pays back on every iteration — and every iteration after that, forever. It's the highest-leverage thing you build in prompt engineering.

EXAM TRAPS

Trap: Refining a prompt against a single hand-picked example. → You'll improve that case and regress others. Build a diverse test set and score the whole thing every iteration.
Trap: Vague success criteria like "make it good." → Unscorable. Criteria must be specific and measurable so an eval can check them automatically.
Trap: Skipping straight to prompt tweaks with no eval. → That's guessing. The three prerequisites — criteria, test, draft — come before tuning.
Trap: Changing five things at once, then not knowing what helped. → Iterate one change at a time against the fixed test set so each delta is attributable.

PRACTICE

Q1 · What are the three prerequisites before prompt engineering?

(1) A clear, specific definition of success criteria; (2) a way to empirically test outputs against them (an eval); and (3) a first-draft prompt to improve. The loop refines a draft against a measurable target — all three must exist first.

Q2 · Why build a diverse test set instead of one example?

Because a prompt change that helps one input can silently break another. A diverse set covering common and edge cases turns each iteration into a full regression check, so improvements are real and regressions surface immediately.

Q3 · What makes success criteria usable in the loop?

They must be specific and measurable — encodable as checks (e.g. "≤ 3 sentences, names all parties, no invented facts") rather than subjective ("good"). Only measurable criteria let you score runs and compare iterations objectively.