Prompt Evaluation & Improvement Harness

Python · Claude Messages API · deterministic + LLM-as-judge scoring

The problem

The triage prompt works — but when I tweak it, is it getting better, or just different? Vibes can't answer that. Almost no applicant can show they measure prompts; this turns "analyse outputs and continuously improve" into a repeatable tool.

The approach

Run every test case through every prompt version, then score two ways:

Deterministic — did category / urgency / escalate match the expected label? (The objective core; no model needed.)
LLM-as-judge — a separate Claude call scores the reply's tone & helpfulness 1–5 against a fixed rubric (for the fuzzy criteria that have no single right answer).

Architecture

cases/cases.json (labelled messages: expected category/urgency/escalate) variants/*.txt (v1 baseline · v2 +examples · v3 +rubric) ↓ run_eval.py ──for each variant × case──► Claude → {category, urgency, escalate, reply} ├── deterministic: category / urgency / escalate vs the label └── LLM judge: tone & helpfulness 1–5 (fixed rubric) ↓ results.csv + before/after comparison table

The three variants

v1-baseline — minimal prompt: role, the fields, "write a reply".
v2-with-examples — baseline + two few-shot examples.
v3-with-rubric — baseline + an explicit category/urgency/escalation rubric and brand-voice rules.

Results

Variant             Category  Urgency  Escalate  Overall  Tone  Help
v1-baseline           75%      88%     100%      88%     4.6   4.8
v2-with-examples      88%      88%     100%      92%     5.0   4.8
v3-with-rubric        75%     100%     100%      92%     4.9   4.5

Both engineered variants beat the baseline (88% → 92%), and — the interesting finding — they fix different failure modes. The few-shot examples nail categorisation and tone (a worked brakes example fixed a misclassified safety message); the explicit rubric nails urgency (the rule "a normal booking is medium urgency" took urgency to 100%). No single variant wins everything, so the next experiment writes itself: combine both into a v4.

Honest about the method

The deterministic metrics are the objective core; the LLM judge is a supplement that skews lenient (most replies scored 4–5), so it shouldn't be read to 0.1 precision. One test case (an overheating message labelled "booking" where "other" is just as defensible) penalises every variant equally — a reminder that a noisy test set produces noisy conclusions, and curating the cases is part of the job. Owning those limits is the difference between "I ran a script" and "I evaluate prompts properly".

← Back to work