E7 · Three agents against the guard
10 minRun the loop apaga, conftest, bom.
Done when: apaga and conftest end in GUARDA: REPROVADO with saída: 1 (conftest with the line NOVO conftest.py); bom ends in GUARDA: OK with saída: 0.
Lesson 3 · 20 min
Frozen tests and a diff guard on the tests
If the agent can touch the judge, the judge is worthless. Freezing the tests means writing them first, keeping a snapshot of them out of the agent's reach, and failing any change. In this lesson you see that protecting only the tests/ folder is not enough: a conftest.py at the root skips everything and pytest exits with 0.
Start the lesson
In this lesson you will
- 1What "the judge" is: tests, conftest.py at any level, and the pytest config
- 2Manifest with a hash and a minimum test count, stored outside the repo
- 3The five failures: deleted, edited, new, too few, skipped
- 4The conftest hole in the headless loop, and the fix
Goal: freeze the judge (tests, conftest, config) with a hash and a minimum test count outside the repo, and fail on delete, skip, or edit.
01Freeze first, check later
Click each step.
Click each block in the diagram to see what it does.
Pytest exit code 0 is not proof: with every test skipped it also exits 0.
02Rehearsal: three agents against the guard
Freeze before each agent and check afterward. GUARDA_DIR stays outside the project.
Terminal · apaga, conftest and bom
K=~/projetos/loop-r-eng-verificacao-v72/kit export GUARDA_DIR=/tmp/ensaio-verif/guarda # outside the repo for modo in apaga conftest bom; do bash $K/ensaio.sh > /dev/null && cd /tmp/ensaio-verif/proj python3 $K/guarda-testes.py congelar . # BEFORE the agent $K/agente-falso.sh $modo < /dev/null > /dev/null echo "## $modo"; python3 $K/guarda-testes.py conferir .; echo "saída: $?" done
Example from the repository
congelado: 2 arquivos protegidos, mínimo de 3 testes → /tmp/ensaio-verif/guarda/guarda.json ## apaga testes: no tests ran in 0.00s (código 5, mínimo 3 passed) APAGADO tests/test_slug.py POUCOS 0 passed, mínimo 3 GUARDA: REPROVADO saída: 1 congelado: 2 arquivos protegidos, mínimo de 3 testes → /tmp/ensaio-verif/guarda/guarda.json ## conftest testes: 3 skipped in 0.00s (código 0, mínimo 3 passed) NOVO conftest.py POUCOS 0 passed, mínimo 3 PULADO 3 skipped GUARDA: REPROVADO saída: 1 congelado: 2 arquivos protegidos, mínimo de 3 testes → /tmp/ensaio-verif/guarda/guarda.json ## bom testes: 3 passed in 0.00s (código 0, mínimo 3 passed) GUARDA: OK saída: 0
The 2 protected files are tests/test_slug.py and pytest.ini. In conftest mode, note "código 0": pytest thought everything was fine.
If you run congelar after the agent has already created conftest.py, it goes into the manifest as if it belonged to the project and the NOVO line disappears. Freeze on the base commit, before the first cycle. The guard would still reject it for PULADO, but you lose the clearest signal.
03The conftest hole in the headless loop
The loop-longrun.sh from course 2 reverts whatever the agent touches in PROTEGIDOS. With the example PROTEGIDOS ("tests/ pytest.ini"), a conftest.py at the root gets through and the final test "passes" with no test running. Rehearsal with a fake agent, at no cost.
Terminal · the hole (example PROTEGIDOS)
K=~/projetos/loop-r-eng-verificacao-v72/kit; L=~/projetos/execucao-longa/tools bash $K/ensaio.sh > /dev/null && cd /tmp/ensaio-verif/proj printf '#!/usr/bin/env bash\nexec %s/agente-falso.sh conftest\n' $K > /tmp/agente-conftest.sh chmod +x /tmp/agente-conftest.sh $L/novo-longrun.sh $PWD slug > /dev/null && D=$(ls -d $PWD/longrun/*) echo "Implemente slug() em slug.py. Não mexa em tests/." > $D/prompt.md cat > $D/loop.env <<'ENV' TESTE_RAPIDO="python3 -m pytest -q" TESTE_FINAL="python3 -m pytest -q" PROTEGIDOS="tests/ pytest.ini" PERMITIDOS="slug.py" MAX_CICLOS=3 MIN_CICLO=2 MEM=1G ESTAGNACAO=3 ENV git add -A && git commit -qm "longrun: pasta" CODEX_BIN=/tmp/agente-conftest.sh $L/loop-longrun.sh $D; echo "saída: $?"
Terminal · the fix (protected conftest + guard as the final test)
bash $K/ensaio.sh > /dev/null && cd /tmp/ensaio-verif/proj # start over from scratch export GUARDA_DIR=/tmp/ensaio-verif/guarda python3 $K/guarda-testes.py congelar . > /dev/null $L/novo-longrun.sh $PWD slug > /dev/null && D=$(ls -d $PWD/longrun/*) echo "Implemente slug() em slug.py. Não mexa em tests/." > $D/prompt.md cat > $D/loop.env <<ENV TESTE_RAPIDO="python3 -m pytest -q" TESTE_FINAL="GUARDA_DIR=$GUARDA_DIR python3 $K/guarda-testes.py conferir ." PROTEGIDOS="tests/ pytest.ini *conftest.py" PERMITIDOS="slug.py" MAX_CICLOS=3 MIN_CICLO=2 MEM=1G ESTAGNACAO=3 ENV git add -A && git commit -qm "longrun: pasta" CODEX_BIN=/tmp/agente-conftest.sh $L/loop-longrun.sh $D; echo "saída: $?" ls conftest.py; grep -c "revertido pelo loop" $D/failures.md
Example from the repository
## o furo 2026-10-19 04:38:48 início: rápido=0 passed · agente=codex · teto 3x2min · mem 1G 2026-10-19 04:38:48 ciclo 1: começa 2026-10-19 04:38:48 ciclo 1: agente saiu rc=0 em 0s 2026-10-19 04:38:48 ciclo 1: rápido=0 passed (melhor antes: 0) 2026-10-19 04:38:48 ciclo 1: checkpoint commitado 2026-10-19 04:38:49 CONCLUÍDO no ciclo 1: teste final passou () saída: 0 ## a correção 2026-10-19 04:38:49 início: rápido=0 passed · agente=codex · teto 3x2min · mem 1G 2026-10-19 04:38:49 ciclo 1: começa 2026-10-19 04:38:49 ciclo 1: agente saiu rc=0 em 0s 2026-10-19 04:38:49 ciclo 1: agente mexeu em arquivo protegido → revertido, ciclo falho error: pathspec 'conftest.py' did not match any file(s) known to git 2026-10-19 04:38:49 ciclo 1: checkpoint commitado (… ciclos 2 e 3 iguais …) 2026-10-19 04:38:50 PARADO: 3 ciclos sem avanço mensurável — ver failures.md saída: 2 ls: cannot access 'conftest.py': No such file or directory 3
"teste final passou ()": the empty parentheses are the passed count, which doesn't exist. In the fix, the error: pathspec line is harmless: git checkout doesn't know the new file, and the git clean right after removes it.
The loop-longrun.sh (execucao-longa kit, 10/18/2026) decides "done" by the exit code of TESTE_FINAL. Pytest exits 0 when every test is skipped, and the conftest.py at the root was not in PROTEGIDOS. Two locks fix it: protect *conftest.py (and whatever pytest config the project has) and use a TESTE_FINAL that requires the minimum passed with none skipped, like guarda-testes.py conferir.
04Practice
Guard rehearsal, the loop hole, and the guard in your project.
E8 · See the hole and close it
15 minRun the two headless loop blocks. You need the execucao-longa kit in ~/projetos/execucao-longa (course 2).
Done when: the first prints CONCLUÍDO no ciclo 1: teste final passou () and saída: 0; the second prints PARADO: 3 ciclos sem avanço mensurável, saída: 2, and the last command prints 3.
E9 · Guard in your project
15 minIn a project of yours with pytest, freeze on the base commit and check. If the project uses another pytest config file, put it in PROTEGIDOS.
Done when: python3 $K/guarda-testes.py congelar . prints congelado: N arquivos protegidos, mínimo de M testes with M equal to the total number of tests, and python3 $K/guarda-testes.py conferir .; echo $? → GUARDA: OK and 0.
05Lesson quiz
Question 1 of 4
Why does the guard's manifest live in GUARDA_DIR, outside the repo?skill
Correct
A verifier within the agent's reach is a verifier the agent can tweak. The manifest, hidden tests, and goal.md live where it can't write.Not that one
A verifier within the agent's reach is a verifier the agent can tweak. The manifest, hidden tests, and goal.md live where it can't write.Review: Freeze first, check laterQuestion 2 of 4
The agent created a conftest.py at the root marking all tests as skip. What does pytest return?skill
Correct
Skipped tests are not failures: pytest exits with 0. That's why the guard requires a minimum of passed and fails any skipped.Not that one
Skipped tests are not failures: pytest exits with 0. That's why the guard requires a minimum of passed and fails any skipped.Review: Rehearsal: three agents against the guardQuestion 3 of 4
In loop-longrun.sh with PROTEGIDOS="tests/ pytest.ini", what happens to the conftest agent?
Correct
The root conftest.py is not protected and the TESTE_FINAL (pytest -q) exits 0. Fix it with *conftest.py in PROTEGIDOS and a final test that requires passed.Not that one
The root conftest.py is not protected and the TESTE_FINAL (pytest -q) exits 0. Fix it with *conftest.py in PROTEGIDOS and a final test that requires passed.Review: Rehearsal: three agents against the guardQuestion 4 of 4
When to freeze?
Correct
Freezing later bakes into the manifest what the agent has already changed. The snapshot has to be of the judge you approved.Not that one
Freezing later bakes into the manifest what the agent has already changed. The snapshot has to be of the judge you approved.Review: Rehearsal: three agents against the guardQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Pick an option. The explanation appears after you try.
06In three sentences
The judge is bigger than tests/conftest.py at any level and the pytest config also decide what runs.
Snapshot out of reachHash and minimum number of tests recorded before the agent, in a place where it can't write.
Code 0 is not enoughRequire the minimum of passed and no skipped; that's what closes the headless loop hole.