Verification and Evals|INEMA.CLUB-PRO

LOOP-R Engineering Track · course 1 of 6 · 8 lessons

Verification and Evals: whoever says it's done can't be the one who did it

A separate, skeptical evaluator, a criterion you can check by command, frozen tests, shortcut detection, a golden set, a calibrated judge, metrics that don't fool you, and a layered gate for your loop. All with small scripts you run first for free, with a fake agent and a fake judge.

Start with lesson 1 See the track area
Night inspection bench with an articulated magnifier over a board, an oscilloscope, and a terminal in the background.
  • 8lessons
  • 22exercises with expected output
  • 14scripts in the kit, all with a no-cost rehearsal
  • 0model calls in the rehearsals

The lessons

In the order you build the gate: separate who checks, check the criterion, protect the judge, hunt shortcuts, measure with evals and a judge, read the numbers, and put it all together.

  1. Lesson 1 · 18 minWhy the agent overrates itself: a separate, skeptical evaluatorThe agent that did the work is the worst judge of it: it writes "3 passed" without having run anything, or it finds the defect and talks itself into approving. In this lesson you separate the one who does from the one who checks: an evaluator that doesn't read the agent's report, runs the test on its own, and starts from the assumption that it's broken.
  2. Lesson 2 · 18 minCommand → output: check the criterion and the /goal evaluatorA criterion only counts if an outsider can check it without trusting the agent. In this lesson you run each criterion of goal.md as a command, see which one passes by shortcut, and learn where the proof has to show up: the /goal evaluator in Claude Code only reads the conversation.
  3. Lesson 3 · 20 minFrozen tests and a diff guard on the testsIf the agent can touch the judge, the judge is worthless. Freezing the tests means writing them first, keeping a snapshot of them out of the agent's reach, and failing any change. In this lesson you see that protecting only the tests/ folder is not enough: a conftest.py at the root skips everything and pytest exits with 0.
  4. Lesson 4 · 20 minShortcuts and reward hacking: how to detect themReward hacking is when the agent meets the letter of the test without doing the work: tampering with the judge, memorizing the answer, silencing the error, or just saying it did it. It has already been measured in coding agents. In this lesson you learn to recognize the four types and set up three detectors that complement each other: hidden tests, diff reading, and the skeptical evaluator.
  5. Lesson 5 · 20 minGolden set: evals for your agentYou changed the prompt, swapped the model, installed a skill: did the agent get better or just different? A golden set is a frozen list of real cases with the right answer, checked by code, that you run on every change. In this lesson you build one, compare two versions, and see a regression hidden behind the same total.
  6. Lesson 6 · 22 minModel judge with a calibrated rubricNot everything becomes a test: "does the message explain why?" takes reading. A model can judge that, as long as the rubric is yes/no, the judgment is repeated, and the judge has been compared with your labels before it counts. In this lesson you calibrate a fake judge (which fails), then a real one, and you see the calibration catch an ambiguous rubric.
  7. Lesson 7 · 18 minHonest metrics: pass@k, pass^k, variance, SWE-bench, and METR"Gets 90% right" can mean it gets it right almost every time, or that it gets it right at least once in three tries. For a loop that runs without you, only the first one will do. In this lesson you measure both on your golden set, see how much the number swings between rounds, and learn to read SWE-bench and the METR horizon without overstating them.
  8. Lesson 8 · 22 minBuild your loop's gateEach lesson gave you one piece: evaluator, criteria, guard, shortcuts, evals, judge. The gate lines them up in a queue, from cheapest to most expensive: whatever the code rejects never gets to spend quota on a model. In this lesson you build the gate, plug it into the headless loop from course 2, and leave with a dated plan for building your own.

Who it's for

  1. 1People who already use Claude Code or Codex in the terminal and have already heard "done, all tests pass" when they didn't.
  2. 2People who will leave a loop running on its own and need a judge the agent can't fool.
  3. 3People who want to compare two versions of an agent with a number, not an impression.

What you'll be able to do

  1. 1Run a separate evaluator that doesn't read the agent's report and fails what only looks done.
  2. 2Freeze the tests and everything pytest reads, and catch the agent that deletes, skips, or memorizes the answer.
  3. 3Build a golden set with regression and a model judge calibrated against your own labels.
  4. 4Read pass@k, pass^k, variance, and benchmarks without being fooled, and close with a layered gate.

Before you start

  1. 1Linux or macOS with git, Python 3, and pytest. The rehearsals don't call any model.
  2. 2Claude Code or Codex signed in through your subscription, only for the steps marked "real".
  3. 3Recommended: course 0 (Base) for the done-criterion scale. Course 2 (Long Execution) uses this gate.