E13 · The hidden regression
10 minRun the rehearsal for both versions.
Done when: v1 prints total 9/10 (90%) and saída: 0; v2 prints REGRESSÃO t06 (passava em v1.tsv) and saída: 1.
Lesson 5 · 20 min
Golden set: evals for your agent
You changed the prompt, swapped the model, installed a skill: did the agent get better or just different? A golden set is a frozen list of real cases with the right answer, checked by code, that you run on every change. In this lesson you build one, compare two versions, and see a regression hidden behind the same total.
Start the lesson
In this lesson you will
- 1The case format: id, category, input, expected, source
- 2The three kinds of grader: code, model, and person
- 3Scoreboard by category and case-by-case regression
- 4When the set got too easy
Goal: build a golden set with real cases, grading by code, and a comparison with the previous version that flags regressions.
01From the real error to the regression verdict
Click each step.
Click each block in the diagram to see what it does.
Code-based grader first; model-based only for what is subjective (lesson 6); human-based to calibrate both.
02Rules for a golden set
From the sources: Anthropic (agent evals), LOOP-R and the FEC track.
1. Three graders
By code (tests, static analysis, checking tool calls), by model (rubric, pairwise comparison, multiple judges) and by person.
The model-based grader has to be calibrated with human labels before it counts.
2. Size
LOOP-R asks for 20 to 50 cases per artifact eval and treats fewer than 30 observations as a weak signal.
The rehearsal uses 10 cases to fit on the screen; yours starts with 20.
3. Freeze with a date
The FEC track's set has a set id, a freeze date, and for each case: id, question, expected facts and category.
In the 05/02/2026 version it announces 30 cases and has 3 filled in: a good structure is not a finished set. Count the lines before trusting the score.
03The set and the rehearsal agent
An issue triage agent: it returns the type (bug, request, question) and whether it is urgent. The 10 cases are examples written for the course.
kit/evals/casos.jsonl (3 of 10 lines; translation for reading; the file in the kit is in Portuguese)
{"id": "t01", "categoria": "bug", "entrada": "After yesterday's deploy, login returns a 500 error for all users.", "espera": {"tipo": "bug", "urgente": true}, "origem": "course example"}
{"id": "t06", "categoria": "pedido", "entrada": "Can you add PDF export?", "espera": {"tipo": "pedido", "urgente": false}, "origem": "course example"}
{"id": "t07", "categoria": "armadilha", "entrada": "The system doesn't work the way I expected: where is the dark mode option?", "espera": {"tipo": "duvida", "urgente": false}, "origem": "course example: looks like a bug, is a question"}kit/evals/prompt-triagem.md (translation for reading; the file in the kit is in Portuguese)
You triage issues for a web system. Read the issue at the end and classify it.
tipo: "bug" (something broken), "pedido" (new feature) or "duvida" (usage question).
urgente: true only if the system is down, money or data is being lost, or there is a security flaw.
Answer with a single line of JSON only, nothing else:
{"tipo": "bug", "urgente": false}Terminal · rehearsal: version 1 against version 2
E=~/projetos/loop-r-eng-verificacao-v72/kit/evals; mkdir -p /tmp/ensaio-evals A="python3 $E/agente-triagem-falso.py" TRIAGEM_VERSAO=1 python3 $E/rodar-evals.py --agente "$A" --saida /tmp/ensaio-evals/v1.tsv; echo "saída: $?" TRIAGEM_VERSAO=2 python3 $E/rodar-evals.py --agente "$A" --saida /tmp/ensaio-evals/v2.tsv \ --base /tmp/ensaio-evals/v1.tsv; echo "saída: $?"
Example from the repository
armadilha 0/1 bug 5/5 duvida 2/2 pedido 2/2 total 9/10 (90%) → /tmp/ensaio-evals/v1.tsv saída: 0 armadilha 1/1 bug 5/5 duvida 2/2 pedido 1/2 total 9/10 (90%) → /tmp/ensaio-evals/v2.tsv MELHOROU t07 REGRESSÃO t06 (passava em v1.tsv) 1 regressão(ões): não promova saída: 1
Both versions score 9/10. Version 2 fixed the trap (t07) and broke a request (t06, which ends in "?" and turned into a question). The total alone would say "tie".
An aggregate score hides a swap: one case gets better, another gets worse, and the number stays the same. Compare case by case with the previous version and treat any regression as a block, even when the total goes up.
04The same set with the real agent
Uses your subscription quota: 10 calls. Each call is a claude -p with no tools.
Terminal · real (continues from the previous block)
python3 $E/rodar-evals.py --agente 'claude -p --tools ""' --saida /tmp/ensaio-evals/real.tsv \ --base /tmp/ensaio-evals/v1.tsv; echo "saída: $?" # ou: --agente 'codex exec -s read-only --skip-git-repo-check -' wc -l < /tmp/ensaio-evals/real.tsv
Example from the repository
armadilha 1/1 bug 5/5 duvida 2/2 pedido 2/2 total 10/10 (100%) → /tmp/ensaio-evals/real.tsv MELHOROU t07 sem regressão saída: 0 11
Claude Code 2.1.303, the subscription's default model, 1 min 19 s for the 10 cases. The model's score may change on your machine; the file with 11 lines (header + 10) will not.
A set that the agent gets entirely right does not separate the good version from the bad one: any change will score 100%. When that happens, add harder cases (the real traps, the ambiguous inputs) until the score has room to be wrong again.
05Practice
Rehearsal, real run and your agent's set.
E14 · The same set with the model
10 minRun the real block with claude -p or codex exec.
Done when: the command ends with a total line and wc -l < /tmp/ensaio-evals/real.tsv → 11.
E15 · Your agent's golden set
40 minChoose an agent or prompt of your own with output you can check. Write 20 cases in the casos.jsonl format, starting with the failures you have already seen, and a prompt that asks for JSON. Run the current version as a baseline.
Done when: wc -l < [your cases].jsonl → 20 or more, and python3 $E/rodar-evals.py --agente "[your agent]" --casos [your cases].jsonl --prompt [your prompt] --saida base.tsv prints the total line with less than 100%.
06Lesson quiz
Question 1 of 4
Version 2 has the same total as version 1. What does rodar-evals.py say?skill
Correct
The comparison is case by case: t06 used to pass and now fails. A regression blocks the promotion even when the total is equal or higher.Not that one
The comparison is case by case: t06 used to pass and now fails. A regression blocks the promotion even when the total is equal or higher.Review: The set and the rehearsal agentQuestion 2 of 4
When should you use a code-based grader instead of a model judge?skill
Correct
Code doesn't drift, has no length bias, and doesn't use your subscription quota. Keep the model judge for what is subjective, and only after it has been calibrated.Not that one
Code doesn't drift, has no length bias, and doesn't use your subscription quota. Keep the model judge for what is subjective, and only after it has been calibrated.Review: From the real error to the regression verdictQuestion 3 of 4
The real agent got 10 out of 10. What do you do with the set?
Correct
A saturated set can't tell versions apart. Harder cases and real traps give you back the power to compare.Not that one
A saturated set can't tell versions apart. Harder cases and real traps give you back the power to compare.Review: The set and the rehearsal agentQuestion 4 of 4
Where do the best cases for a golden set come from?
Correct
A real failure is the case you need to make sure never comes back. Each case carries its origin so someone knows why it exists.Not that one
A real failure is the case you need to make sure never comes back. Each case carries its origin so someone knows why it exists.Review: From the real error to the regression verdictQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Pick an option. The explanation appears after you try.
07In three sentences
Real cases, frozenOne JSON line per case, with the expected result and origin; changing a case means a new version of the ruler.
Code gradesAsk for JSON and check it field by field; use a model judge only for the subjective.
Regression blocksCompare case by case with the previous version; an equal total can hide a swap.