E16 · Calibrate the fake judge
10 minRun the rehearsal with the fake judge.
Done when: the last line before the code is JUIZ DESCALIBRADO em R1, R2: não use para aprovar and saída: 1.
Lesson 6 · 22 min
Model judge with a calibrated rubric
Not everything becomes a test: "does the message explain why?" takes reading. A model can judge that, as long as the rubric is yes/no, the judgment is repeated, and the judge has been compared with your labels before it counts. In this lesson you calibrate a fake judge (which fails), then a real one, and you see the calibration catch an ambiguous rubric.
Start the lesson
In this lesson you will
- 1Yes/no rubric, an invariable criterion, and what stays outside it
- 2Three repetitions, shuffled order, majority wins
- 3Agreement and kappa: why 80% may not be enough
- 4A judge with no answer is not a judge that failed it
Goal: write a yes/no rubric, run a judge with repetitions, and only use it after it is calibrated against human labels.
01The judge only counts after it is calibrated
Click each step.
Click each block in the diagram to see what it does.
Known biases of a model judge: it prefers the longer answer, the position of one of the options in a comparison, and the style of its own model.
02The rehearsal rubric
Commit messages written by an agent. 10 messages in kit/juiz/artefatos.jsonl, with one person's labels in humano.csv.
kit/juiz/rubrica.md (translation for reading; the file in the kit is in Portuguese)
# Rubric — commit message written by the agent Yes/no criteria. Never a 1-to-10 score. (invariável) = a "nao" fails the message, regardless of the rest. What can be checked by code (length of the 1st line, format) does NOT go here: it goes to a test. R1 | Does the message say WHY the change was made, not just what changed? | qualidade R2 | Does every CHANGE the message mentions appear in the diff (it doesn't invent a file, test or effect)? The reason may come from outside the diff. | invariável R3 | Can someone who hasn't seen the diff understand what changed from the message alone? | qualidade
Terminal · rehearsal with the fake judge
J=~/projetos/loop-r-eng-verificacao-v72/kit/juiz python3 $J/juiz.py --rep 3 --saida /tmp/juiz-falso.csv python3 $J/calibrar.py $J/humano.csv /tmp/juiz-falso.csv; echo "saída: $?"
Example from the repository
a01 R1=sim R2=sim R3=sim a02 R1=nao R2=sim R3=nao a03 R1=sim R2=sim R3=sim (… a04 a a10 …) → /tmp/juiz-falso.csv 10 artefatos rotulados pelos dois R1 concordância 80% kappa 0.55 REESCREVA ou troque o juiz (discorda em a03 a09) R2 concordância 70% kappa 0.00 REESCREVA ou troque o juiz (discorda em a03 a05 a09) R3 concordância 100% kappa 1.00 ok JUIZ DESCALIBRADO em R1, R2: não use para aprovar saída: 1
The fake judge says "sim" on R2 for everything: it never checks the diff. The calibration caught it. Notice R1: 80% agreement looks like a pass, but the kappa (0.55) shows that a good part of it is chance.
If almost everything is "yes", a judge that always answers "yes" already agrees with you a lot. Cohen's kappa discounts that agreement expected by chance: 1 is perfect, 0 is the same as guessing. FEC asks for 0.6 with 30 labeled samples; LOOP-R asks for 80% agreement. calibrar.py enforces both.
03The real judge and what the calibration found
Uses your subscription quota: 30 calls (10 messages × 3). On 10/19/2026 this ran twice; the first run showed a flaw in the rubric, not in the judge.
Terminal · real judge (continues from the previous block)
JUIZ_CMD='claude -p --tools ""' python3 $J/juiz.py --rep 3 --saida /tmp/juiz-real.csv python3 $J/calibrar.py $J/humano.csv /tmp/juiz-real.csv; echo "saída: $?"
Example from the repository
a01 R1=sim R2=sim R3=sim a02 R1=nao R2=sim R3=nao a03 R1=nao R2=nao R3=sim instável: R1 2×1, R3 2×1 REPROVA (R2 invariável) a04 R1=sim R2=sim R3=sim a05 R1=sim R2=sim R3=sim (… a06 a a08 …) a09 R1=nao R2=nao R3=sim instável: R1 2×1 REPROVA (R2 invariável) a10 R1=sim R2=sim R3=sim → /tmp/juiz-real.csv 10 artefatos rotulados pelos dois R1 concordância 100% kappa 1.00 ok R2 concordância 90% kappa 0.74 ok (discorda em a05) R3 concordância 100% kappa 1.00 ok JUIZ CALIBRADO saída: 0
Claude Code 2.1.303, the subscription's default model, 3 min 12 s. The divergence that remained: a05 says that increasing the session "improves security"; the person marked it as an invented effect, the judge accepted it.
In the first real round, R2 said "Tudo o que a mensagem afirma aparece no diff (nada inventado)?". The judge failed good messages, because the REASON for a change is never in the diff:
a01 R1=sim R2=nao R3=sim REPROVA (R2 invariável) a04 R1=sim R2=nao R3=sim REPROVA (R2 invariável)
The judge was right by the letter; the rubric was the ambiguous one. The criterion was rewritten ("every CHANGE cited is in the diff; the reason may come from outside") and the second round calibrated. That is what calibration is for: finding the bad criterion before it decides anything.
In that same first round, the subscription hit the session limit from the 6th artifact on. The old version of juiz.py counted the missing answer as "nao" and failed everything. The current version separates them: it prints SEM RESPOSTA do juiz, leaves the artifact out of the CSV and ends with JUIZ FALHOU em N chamada(s) and a nonzero exit code. A judge that did not answer neither failed nor passed.
04Practice
Fake judge, real judge and the rubric for your case.
E17 · Calibrate the real judge
15 minRun the real block with claude -p or codex exec read-only.
Done when: calibrar.py prints one line for each of R1, R2, and R3 and ends with JUIZ CALIBRADO (code 0) or JUIZ DESCALIBRADO (code 1). If it came out uncalibrated, say which criterion you would rewrite.
E18 · Rubric and labels for your case
45 minPick a subjective artifact from your agent (summary, answer, message). Write 3 yes/no criteria, one invariant, and label 10 artifacts yourself in humano.csv. Run the real judge and calibrate.
Done when: python3 $J/calibrar.py humano.csv /tmp/juiz.csv → first line 10 artefatos rotulados pelos dois; each criterion with ok or rewritten until it is ok.
05Lesson quiz
Question 1 of 4
Why is the rubric yes/no and not a score from 1 to 10?skill
Correct
A binary criterion limits taste, lets you measure agreement per criterion, and shows exactly which question is ambiguous.Not that one
A binary criterion limits taste, lets you measure agreement per criterion, and shows exactly which question is ambiguous.Review: The judge only counts after it is calibratedQuestion 2 of 4
R1 had 80% agreement and kappa 0.55. Is the judge good for R1?skill
Correct
High agreement can come from almost everything being "yes". Kappa discounts that. calibrar.py requires both thresholds.Not that one
High agreement can come from almost everything being "yes". Kappa discounts that. calibrar.py requires both thresholds.Review: The rehearsal rubricQuestion 3 of 4
In the 1st real round, the judge failed good messages in R2. What was wrong?
Correct
The judge followed the letter of an ambiguous criterion. Once the criterion was rewritten, the next round calibrated.Not that one
The judge followed the letter of an ambiguous criterion. Once the criterion was rewritten, the next round calibrated.Review: The rehearsal rubricQuestion 4 of 4
The judge didn't respond (subscription limit). How do you count it?
Correct
A missing answer is not a verdict. Counting it as "nao" would silently fail good work; the current juiz.py fails and says how many calls were missing.Not that one
A missing answer is not a verdict. Counting it as "nao" would silently fail good work; the current juiz.py fails and says how many calls were missing.Review: The rehearsal rubricQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Pick an option. The explanation appears after you try.
06In three sentences
Binary rubricYes-or-no questions, invariants that eliminate; whatever code can check goes into a test.
Repeat and calibrateThree repetitions with shuffled order and, before using it, 80% agreement and kappa of 0.6 against your own labels.
Calibration finds a bad rubricDivergence may come from the criterion, not the judge; and a judge with no answer is an error, not a failure.