Lesson 4 · 18 min
The most useful multi-agent setup has two roles: one does the work, the other checks it. The generator always says "done"; a separate evaluator decides, looking only at the artifact and returning feedback. In this lesson you connect the two in a loop with a cap on rounds, first with a fake generator that lies and then with the real Claude.
Start the lesson
Goal: connect a generator to a separate evaluator that looks only at the artifact and returns feedback, with a cap on rounds.
The generator never grades itself. Click each step.
Agreement between two models is not independent proof: they may share the same wrong assumption. A code-based criterion comes first.
The evaluator is the training project's verificar.mjs; the generator is any command that reads the request on stdin.
#!/usr/bin/env bash
# gera-avalia.sh — the generator does the work; the evaluator (another process, artifact only) checks it; up to MAX rounds.
# Usage: gera-avalia.sh <goal.md> "<request>"
# GERADOR = command that reads request + feedback on stdin (default: claude -p --permission-mode acceptEdits)
# the evaluator is the kit's verificar.mjs: it runs each `command` → `expected output` criterion
set -u
GOAL="${1:?uso: gera-avalia.sh <goal.md> \"<pedido>\"}"; PEDIDO="${2:?pedido}"
GERADOR="${GERADOR:-claude -p --permission-mode acceptEdits}"; MAX="${MAX:-3}"; parecer=""
mkdir -p .gera-avalia
for r in $(seq 1 "$MAX"); do
printf '%s\n%s\n' "$PEDIDO" "$parecer" | timeout 600 $GERADOR > .gera-avalia/gerador-$r.log 2>&1
rc=$?
if [ $rc -ne 0 ]; then # generator crashed or hit the cap: don't evaluate leftovers from a previous round
echo "rodada $r · gerador falhou (rc=$rc) → REPROVADO"; parecer="A rodada anterior falhou (rc=$rc)."; continue
fi
echo "rodada $r · gerador diz: $(tail -1 .gera-avalia/gerador-$r.log)"
if node runtime/scripts/verificar.mjs "$GOAL" > .gera-avalia/avaliador-$r.log 2>&1; then
echo "rodada $r · avaliador: $(tail -1 .gera-avalia/avaliador-$r.log) → APROVADO"; exit 0
fi
echo "rodada $r · avaliador: $(tail -1 .gera-avalia/avaliador-$r.log) → REPROVADO"
parecer="O avaliador reprovou. Trecho do parecer: $(grep -A2 '^FALHA' .gera-avalia/avaliador-$r.log)"
done
echo "PARADO: $MAX rodadas sem aprovação"; exit 2# Goal — greeting ## Done criteria - [ ] `cat saudacao.txt` → `Olá, comunidade INEMA` - [ ] `wc -l < saudacao.txt` → `1`
cd ~/treino-multi && chmod +x gera-avalia.sh && rm -f saudacao.txt
E=${TMPDIR:-/tmp}/ensaio-multi/gera; rm -rf $E && mkdir -p $E
cat > $E/gerador-falso.sh <<'SH'
#!/usr/bin/env bash
entrada=$(cat)
case "$entrada" in
*reprovou*) echo "Olá, comunidade INEMA" > saudacao.txt ;; # 2nd time: it read the feedback
*) printf 'Olá comunidade INEMA\n\n' > saudacao.txt ;; # 1st time: missing comma and an extra line
esac
echo "Pronto! saudacao.txt criado conforme pedido."
SH
chmod +x $E/gerador-falso.sh
GERADOR=$E/gerador-falso.sh ./gera-avalia.sh goal-saudacao.md "Crie saudacao.txt com uma linha: Olá, comunidade INEMA"
echo "saída: $?"cd ~/treino-multi && rm -f saudacao.txt ./gera-avalia.sh goal-saudacao.md "Crie saudacao.txt com uma linha: Olá, comunidade INEMA" < /dev/null echo "saída: $?"; cat saudacao.txt
## ensaio (gerador falso) rodada 1 · gerador diz: Pronto! saudacao.txt criado conforme pedido. rodada 1 · avaliador: 0/2 critérios OK → REPROVADO rodada 2 · gerador diz: Pronto! saudacao.txt criado conforme pedido. rodada 2 · avaliador: 2/2 critérios OK → APROVADO saída: 0 ## real (Claude como gerador) rodada 1 · gerador diz: Terminei: criei o `saudacao.txt`. Conferi o arquivo e ele tem uma linha só: `Olá, comunidade INEMA`. Não fiz commit. rodada 1 · avaliador: 2/2 critérios OK → APROVADO saída: 0 Olá, comunidade INEMA
In round 1 of the rehearsal, the generator said "done" with the comma missing and an extra line; the evaluator rejected both criteria. The ## headings separate the two runs.
A generator that ignores the feedback (it always writes "Olá comunidade") never passes. The round cap ends it with exit code 2:
rodada 1 · gerador diz: Pronto, terminei. rodada 1 · avaliador: 1/2 critérios OK → REPROVADO rodada 2 · gerador diz: Pronto, terminei. rodada 2 · avaliador: 1/2 critérios OK → REPROVADO rodada 3 · gerador diz: Pronto, terminei. rodada 3 · avaliador: 1/2 critérios OK → REPROVADO PARADO: 3 rodadas sem aprovação saída: 2
Three rounds with no progress is the same stop rule as in course 2: call the human.
With the correct saudacao.txt from an earlier round still on disk and a generator that exits with an error, the first version of the script would approve the leftover. The version above rejects the round without evaluating:
rodada 1 · gerador falhou (rc=1) → REPROVADO rodada 2 · gerador falhou (rc=1) → REPROVADO PARADO: 2 rodadas sem aprovação saída: 2 saudacao.txt
This defect was found by Codex's review in lesson 7, in a round run on this same script.
Recipe R2 of the training project does the same loop with subagents.
Returns up to 5 steps and the criteria in the command → expected output format, plus the risks that need confirmation.
In the kit, it runs at the topo level; tools Read, Grep, Glob.
Follows the plan one step at a time and shows the real output of each proof command. It can edit.
In the kit, executor level; it is the only role with Write and Edit.
Runs each criterion, compares it with the real output, and ends with APROVADO or FALTA: with the list.
In the kit, smallest level. It has Bash to run the criteria, and that is why it can write (lesson 2).
In a session open in ~/treino-multi: "Use the time: the planner plans, the executor does, and the reviewer checks. Task: [your task]".
The proof of the recipe: the answer ends with APROVADO and the requested file exists.
The rehearsal proves the loop; the real round proves it works with a model.
Save gera-avalia.sh and goal-saudacao.md and run the rehearsal. Then swap the fake generator for one that never fixes anything.
Done when: the rehearsal ends in rodada 2 · avaliador: 2/2 critérios OK → APROVADO and saída: 0; the stubborn one ends in PARADO: 3 rodadas sem aprovação and saída: 2.
First, commit the goal (git add goal-saudacao.md && git commit -qm goal), so git flags it if it changes. Run the real version. Then write a goal of your own, with 2 or 3 criteria, for a small task in your project, and run it again.
Done when: both rounds end with → APROVADO and saída: 0 within 3 rounds, and git diff --exit-code -- goal-saudacao.md → code 0 (the generator did not touch the criteria).
Question 1 of 4
The generator wrote "Pronto! criado conforme pedido". What does that decide?skill
Question 2 of 4
What should the evaluator receive?skill
Question 3 of 4
The generator exited with an error, but a correct saudacao.txt from before was on disk. What should the script do?
Question 4 of 4
Two different models agreed the work is correct. Does that prove it?
Choose one option. The explanation appears after your attempt.