E7 · Redo the verdicts
10 minRun the first block.
Done when: E0001 → VEREDITO: A_SEGUE with FALHA on reclamou and saída: 1; E0002 → VEREDITO: B_GANHOU, z=2.8543 and saída: 0.
Lesson 3 · 20 min
Decision card, promotion and rollback
A keep in the lab is not the official version yet. In production, the new version only goes up when the verdict says B won without making anything worse, someone approves it from the five-line card, and there is a way back that deletes nothing. In this lesson you redo the published verdicts from the LOOP-R example, promote, and roll back.
Start the lesson
In this lesson you will
- 1Verdict: B_GANHOU, A_SEGUE or AMOSTRA_INSUFICIENTE
- 2The five-line decision card
- 3Promote: verdict + approval, with a commit
- 4Rollback: the link goes back, the folder stays
Goal: read an A/B verdict with guardrails, show the 5-line card, promote only with approval, and roll back without deleting the version.
01From verdict to official version, and back
The decidir, promover and reverter commands of the LOOP-R runner. Click each block.
Rollback is not undoing the commit: it is a new commit that changes the link. The whole history stays in git.
Runner invariants: never promote without B_GANHOU + approval, never delete a cycle, a version, or a memory line, never run two experiments at the same time.
02The two published verdicts from the example
The LOOP-R vendas-whatsapp example uses SYNTHETIC data (a generator with seeds chosen to reflect designed effects) and simulated human decisions; the agents ran for real. The kit redoes the math on a copy.
Terminal · cost-free rehearsal
K=~/projetos/loop-r-eng-aprende-v72/kit
X=~/projetos/loop-r/exemplos/vendas-whatsapp # git clone https://github.com/inematds/loop-r ~/projetos/loop-r (SYNTHETIC data)
E=${TMPDIR:-/tmp}/ensaio-aprende/a3; rm -rf $E; L=$E/loop # a COPY: nothing is written to the loop-r repo
python3 $K/loopr.py iniciar $L --v1 $X/versoes/v1/executor.prompt.md --candidata $X/versoes/v2/executor.prompt.md
echo "## E0001 · H2 (até 80 palavras), tolerância 0 em reclamações"
python3 $K/loopr.py veredito $L --csv $X/dados/execucoes.csv --de 2026-02-02 --ate 2026-04-26 \
--metrica resposta --minimo 166 --vigiar reclamou:0 --vigiar optout:0.005; echo "saída: $?"
echo "## E0002 · H3 (abrir citando um ponto da avaliação)"
python3 $K/loopr.py veredito $L --csv $X/dados/execucoes.csv --de 2026-05-04 --ate 2026-07-26 \
--metrica resposta --minimo 212 --vigiar reclamou:0.01 --vigiar optout:0.005; echo "saída: $?"Example from the repository
loop iniciado: versoes/atual → v1 ## E0001 · H2 (até 80 palavras), tolerância 0 em reclamações VEREDITO: A_SEGUE Métrica do teste: resposta A=0.1933 (58/300) B=0.2967 (89/300) z=2.9426 p=0.0033 Vigiada reclamou: A 1/300 vs B 6/300, Δ=+0.0167 tol 0 → FALHA Vigiada optout: A 0/300 vs B 1/300, Δ=+0.0033 tol 0.005 → DENTRO saída: 1 ## E0002 · H3 (abrir citando um ponto da avaliação) VEREDITO: B_GANHOU Métrica do teste: resposta A=0.1933 (58/300) B=0.2933 (88/300) z=2.8543 p=0.0043 Vigiada reclamou: A 2/300 vs B 4/300, Δ=+0.0067 tol 0.01 → DENTRO Vigiada optout: A 3/300 vs B 1/300, Δ=-0.0067 tol 0.005 → DENTRO saída: 0
The same numbers as the published ledger: E0001 with 58/300 against 89/300 (z=2.94) was discarded because of complaints (6 against 1, tolerance 0); E0002 with 58/300 against 88/300 (z=2.85, p=0.004) won. Even with the 0.01 tolerance the example adopted later, E0001 would still fail (Δ=0.0167).
Reading only the test metric is not enough. E0001 had a bigger gain than E0002 and still could not go up: a guardrail went past the tolerance. And the tolerance is agreed before; changing it after seeing the number is rewriting the hypothesis.
03Card, promotion, and rollback
Continues the previous block. The card's cost is the sum of the cost.md of cycle 0004 of the example (5 agents, ~392 thousand tokens); the example also records it in reais, the course counts in tokens.
Terminal · cost-free rehearsal
K=~/projetos/loop-r-eng-aprende-v72/kit
E=${TMPDIR:-/tmp}/ensaio-aprende/a3; L=$E/loop # continues the previous block (verdict B_GANHOU recorded)
python3 $K/loopr.py promover $L; echo "saída: $?"
python3 $K/loopr.py cartao $L --hipotese "H3 — abrir citando a avaliação" --tokens 392000
python3 $K/loopr.py decidir $L aprovado --por "Seu Nome"
python3 $K/loopr.py promover $L; echo "atual → $(readlink $L/versoes/atual)"
python3 $K/loopr.py reverter $L --motivo "resposta caiu no 1º ciclo após a promoção"
echo "atual → $(readlink $L/versoes/atual) · pastas: $(ls $L/versoes | tr '\n' ' ')"
git -C $L log --format=%sExample from the repository
RECUSADO: promover exige veredito B_GANHOU e decisão aprovado (veredito: B_GANHOU; decisão: nenhuma) saída: 1 CICLO — hipótese: "H3 — abrir citando a avaliação" Resultado: resposta 0.1933 → 0.2933 ((58/300) vs (88/300), z=2.8543, p=0.0043) [VEREDITO: B_GANHOU] Vigiado: reclamou ok · optout ok Custo: 392.000 tokens neste ciclo (sai da cota) Decisão: aprovar / rejeitar / esperar mais dados decisão gravada: aprovado PROMOVIDA: versoes/atual → v2 atual → v2 ROLLBACK: versoes/atual → v1 (a pasta v2 continua lá) atual → v1 · pastas: atual v1 v2 loop-r: rollback v2 → v1 loop-r: promote v2 loop-r: iniciar
The first attempt to promote is refused: good verdict, but nobody approved. After the rollback, atual points to v1 and both folders are still there.
04The owner wants to promote now
Simulated conversation. Version B's test is at half the sample. Choose what you answer.
Martaclinic owner (fictional character) · L1 approver
Pick your answer. The comment shows up right after, and the conversation continues based on your choice.
05Practice
Everything runs on a copy in /tmp: the kit never writes to the loop-r repository.
E8 · Promote and roll back
10 minRun the second block.
Done when: the first line starts with RECUSADO; then atual → v2, and at the end atual → v1 · pastas: atual v1 v2; git -C $L log --oneline | wc -l → 3.
E9 · Your test card
20 minFor the lesson 1 hypothesis, write the tolerances for each "do not worsen" and the minimum N before you run. For a rare event, use at least 2 standard errors of the base rate in the planned sample (loop-r/docs/05 §2). Commit hipotese.md with the tolerances before the next cycle.
Done when: H=$(git log --diff-filter=A --format=%ct -- hipotese.md | tail -1); C=$(git log --reverse --format=%ct --grep='^keep:' | head -1); [ -n "$H" ] && { [ -z "$C" ] || [ "$H" -lt "$C" ]; }; echo "saída: $?" → saída: 0 (hipotese.md entered git before the first keep), and the committed hipotese.md has a numeric tolerance for each "do not worsen".
06Lesson quiz
Question 1 of 4
B raises the response rate from 19% to 30% with p=0.003, but complaints go past the tolerance. What is the verdict?skill
Correct
B only wins if it improves the test metric and no "do not worsen" goes past the tolerance. That was the case for E0001 in the example.Not that one
B only wins if it improves the test metric and no "do not worsen" goes past the tolerance. That was the case for E0001 in the example.Review: The two published verdicts from the exampleQuestion 2 of 4
The verdict was B_GANHOU and the approver did not respond by the deadline. What happens at L1?skill
Correct
Silence is not approval. Without decisao.md = approved, the runner refuses to promote.Not that one
Silence is not approval. Without decisao.md = approved, the runner refuses to promote.Review: From verdict to official version, and backQuestion 3 of 4
In the rollback from v2 to v1, what happens to the v2 folder?
Correct
Rollback changes the versoes/atual link and records the reason. Never deleting a version is a runner invariant.Not that one
Rollback changes the versoes/atual link and records the reason. Never deleting a version is a runner invariant.Review: From verdict to official version, and backQuestion 4 of 4
When is it allowed to stop a test before the minimum sample?
Correct
The only early stop is by guardrail. Stopping for a partial lead promotes noise.Not that one
The only early stop is by guardrail. Stopping for a partial lead promotes noise.Review: The owner wants to promote nowQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Choose an answer. The explanation appears after you try.
07In three sentences
VerdictB wins with significance, minimum N and no guardrail outside the tolerance; otherwise A stays.
ApprovalAt L1, the five-line card goes to the human; without "aprovado", nothing goes up.
BackRollback is a commit that changes the link and notes the reason; no version is deleted.