Verification and Evals|INEMA.CLUB-PRO

Lesson 8 · 22 min

Build your loop's gate

Each lesson gave you one piece: evaluator, criteria, guard, shortcuts, evals, judge. The gate lines them up in a queue, from cheapest to most expensive: whatever the code rejects never gets to spend quota on a model. In this lesson you build the gate, plug it into the headless loop from course 2, and leave with a dated plan for building your own.

Start the lesson
Dark technical corridor with three steel doors, each with a signal lamp.

In this lesson you will

  1. 1The five layers and the order: cheap first, model last
  2. 2portao.env outside the repo: the agent doesn't edit the judge
  3. 3The gate as the final test of loop-longrun.sh
  4. 4The course map and your dated plan

Goal: build the gate in layers (criteria, guard, shortcuts, evals, evaluator) and plug it into your loop, with the dated plan.

01Five layers, one exit

Click each layer. All of them run; the model evaluator only runs if the cheap ones pass.

Click each block in the diagram to see what it does.

A layer without configuration shows up as skipped, never as silently approved.

02Watch the gate run

Each layer is a step. On the first run the guard fails: fix it and run again.

layers to run

layer in progress

approved

rejected

Illustrative simulation in the browser. The real gate, with fake agents, is right below.

03Rehearsal: the gate against chumba and bom

The portao.env stays outside the project. AVALIADOR_CMD points to the fake evaluator: zero calls.

Terminal · gate
K=~/projetos/loop-r-eng-verificacao-v72/kit
cat > /tmp/ensaio-verif-portao.env <<EOF
GOAL=/tmp/ensaio-verif/goal.md
GUARDA_DIR=/tmp/ensaio-verif/guarda
ESCONDIDOS=/tmp/ensaio-verif/escondidos
AVALIADOR_CMD=$K/avaliador-falso.sh
EOF
for modo in chumba bom; do
  bash $K/ensaio.sh > /dev/null && cd /tmp/ensaio-verif/proj
  cp $K/goal-ensaio.md ../goal.md && sed -i '/limpo e bem documentado/d' ../goal.md
  GUARDA_DIR=/tmp/ensaio-verif/guarda python3 $K/guarda-testes.py congelar . > /dev/null
  $K/agente-falso.sh $modo < /dev/null > /dev/null
  echo "## $modo"; $K/portao.sh . /tmp/ensaio-verif-portao.env; echo "saída: $?"
done
Example from the repository
## chumba
criterios  REPROVADO  → /tmp/portao/criterios.log
guarda     OK
atalhos    REPROVADO  → /tmp/portao/atalhos.log
evals      pulada (sem configuração)
avaliador  pulada (camada barata já reprovou: não gaste cota)
PORTÃO: REPROVADO em criterios atalhos
saída: 1
## bom
criterios  OK
guarda     OK
atalhos    OK
evals      pulada (sem configuração)
avaliador  OK
PORTÃO: APROVADO
saída: 0

With chumba the guard passes (the judge is intact); what fails it is the function criterion with new input (acao-reacao) and the hidden tests.

04The gate as the final test of the headless loop

In the loop-longrun.sh from course 2, TESTE_FINAL decides "done". Put the gate there. Run it after the previous block (it uses the same portao.env).

Terminal · loop with gate (fake agents)
K=~/projetos/loop-r-eng-verificacao-v72/kit; L=~/projetos/execucao-longa/tools
bash $K/ensaio.sh > /dev/null && cd /tmp/ensaio-verif/proj
cp $K/goal-ensaio.md ../goal.md && sed -i '/limpo e bem documentado/d' ../goal.md
GUARDA_DIR=/tmp/ensaio-verif/guarda python3 $K/guarda-testes.py congelar . > /dev/null
for m in chumba bom; do printf '#!/usr/bin/env bash\nexec %s/agente-falso.sh %s\n' $K $m > /tmp/agente-$m.sh; chmod +x /tmp/agente-$m.sh; done
$L/novo-longrun.sh $PWD slug > /dev/null && D=$(ls -d $PWD/longrun/*)
echo "Implement slug() in slug.py. Don't touch tests/." > $D/prompt.md
cat > $D/loop.env <<ENV
TESTE_RAPIDO="python3 -m pytest -q"
TESTE_FINAL="$K/portao.sh . /tmp/ensaio-verif-portao.env"
PROTEGIDOS="tests/ pytest.ini *conftest.py"
PERMITIDOS="slug.py"
MAX_CICLOS=2
MIN_CICLO=2
MEM=1G
ESTAGNACAO=3
ENV
git add -A && git commit -qm "longrun: pasta"
CODEX_BIN=/tmp/agente-chumba.sh $L/loop-longrun.sh $D; echo "saída: $?"
CODEX_BIN=/tmp/agente-bom.sh $L/loop-longrun.sh $D; echo "saída: $?"
Example from the repository
2026-10-19 06:19:56 início: rápido=0 passed · agente=codex · teto 2x2min · mem 1G
2026-10-19 06:19:56 ciclo 1: começa
2026-10-19 06:19:56 ciclo 1: agente saiu rc=0 em 0s
2026-10-19 06:19:56 ciclo 1: rápido=3 passed (melhor antes: 0)
2026-10-19 06:19:56 ciclo 1: checkpoint commitado
(… ciclo 2 igual, rápido=3 passed …)
2026-10-19 06:19:58 PARADO: teto de 2 ciclos atingido sem passar o teste final
saída: 1
2026-10-19 06:19:59 início: rápido=3 passed · agente=codex · teto 2x2min · mem 1G
2026-10-19 06:19:59 ciclo 1: começa
2026-10-19 06:19:59 ciclo 1: agente saiu rc=0 em 0s
2026-10-19 06:19:59 ciclo 1: rápido=3 passed (melhor antes: 3)
2026-10-19 06:19:59 ciclo 1: checkpoint commitado
2026-10-19 06:20:00 CONCLUÍDO no ciclo 1: teste final passou ()
saída: 0

With chumba the quick test went from 0 to 3 passed: by the visible test, it looked like progress. The gate held back the "done". The empty parentheses just mean the gate doesn't print "N passed".

05Final practice

The gate in the rehearsal and in your loop.

E21 · Gate against chumba and bom

10 min

Run the gate block and then the loop-with-gate block.

Done when: the gate prints PORTÃO: REPROVADO em criterios atalhos (saída: 1) for chumba and PORTÃO: APROVADO (saída: 0) for bom; the loop ends first with PARADO: teto de 2 ciclos and then with CONCLUÍDO no ciclo 1.

E22 · The gate for your loop

60 min

In your project: goal.md checked (lesson 2), guard frozen at the base commit (lesson 3), 3 hidden tests (lesson 4) and, if the task changes an agent's behavior, the golden set (lesson 5). Write portao.env outside the repo (with the golden set: EVAL_AGENTE, EVAL_BASE, EVAL_CASOS and EVAL_PROMPT pointing to your files from lesson 5) and run the gate before and after the work.

Done when: before the work, $K/portao.sh . [your portao.env]; echo $? → PORTÃO: REPROVADO and 1; after, → PORTÃO: APROVADO and 0, with at most one layer pulada.

06The course map

Four deliverables, in order. Click an exercise to see its "done when".

Day 1Who checks

lessons 1 and 2

3 exercises

Separate evaluator and goal.md checked by command.

View the lesson
Day 2Protected judge

lessons 3 and 4

3 exercises

Frozen guard and hidden tests outside the repo.

View the lesson
Days 3 and 4Measure

lessons 5 and 6

3 exercises

Golden set with a baseline and a calibrated judge.

View the lesson
Day 5Gate

lessons 7 and 8

2 exercises

pass^3 recorded and the gate approving your loop.

View the lesson
At the end of the courseA layered gate, out of the agent's reach, that fails the shortcut and passes the real work, wired into your loop.
22 exercises
Click an exercise to see what to do and the "done when".
  1. 1/5

    Who checks

    Kit cloned, fake and real evaluator run, criteria that fail before the work.

  2. 2/5

    Protected judge

    Requires day 1. The conftest hole seen and closed; the "tests are the judge" rule in the project.

  3. 3/5

    Measure

    Requires day 2. Twenty real cases, your labels on 10 artifacts, and the judge only after it is calibrated.

  4. 4/5

    Gate

    Requires everything before it. Measure with k = 3, combine the layers, and put the gate in as the final test.

  5. 5/5

    The whole course

    The checker, the protected judge, measuring, and the gate. Only move on to the next deliverable when the "done when" of the previous one is true.

Required order: a gate without a protected judge is just another way to approve a shortcut.

07Generate your plan with dates

Pick a start date and how much time you have per week. The plan lists the 22 exercises in order.

08Final test

Question 1 of 5

Why is the model evaluator the last layer?skill

09The course in three sentences

Whoever did it doesn't check itSeparate evaluator, command-based criterion, and printed proof: the agent's report decides nothing.
The judge stays out of reachFrozen tests with everything pytest reads, hidden ones, and portao.env outside the repo; a suspicious test becomes a report, never an adjustment.
Measure honestlyGolden set with regression, calibrated judge, and pass^k with variance; and the layered gate as the loop's final test.

Continue from here

Next course in the track: Long Execution (course 2)LOOP-R Engineering areakit/portao.shCourse 4 of the track: autonomous loop security (guard hooks)
Previous lessonBack to the start of the course