E1 · Fail and pass
5 minClone the course repository and run the rehearsal above.
Done when: the gate ends with HIPÓTESE: REPROVADA and saída: 1; the triage with HIPÓTESE: OK and saída: 0; the conversion shows 61 semanas and the AVISO.
Lesson 1 · 15 min
A hypothesis with a number
A loop that learns starts with a sentence that can turn out wrong. "Improve the triage" is never wrong, so it teaches nothing. In this lesson you write the hypothesis with process, metric, current, target, measurement command, and what must not get worse, and you work out how long the test takes before you run it.
Start the lesson
In this lesson you will
- 1The seven fields of a testable hypothesis
- 2Three metric speeds: money, signal, and artifact
- 3Minimum sample: how many weeks the test takes
- 4Why 3% → 5% is almost never testable directly
Goal: write a hypothesis with a metric, current value, target, measurement command, and do-not-worsen, and work out how long the test takes before you run it.
01From a wish to a testable hypothesis
This is the first question of /loop-r iniciar: which process, and which number you want to move from what to what. Click each block.
The hypothesis is written before the experiment, in a file. After the result, nobody rewrites the target.
Fields and math come from loop-r/.claude/skills/loop-r/SKILL.md (start) and loop-r/docs/05 (measurement).
02Three metric speeds
LOOP-R proposes one of each and tests on the fastest one that still has a plausible link to the money one (loop-r/docs/05 §1).
1. Money
Conversion, ticket, margin. It's what matters, but the one who answers is the customer, weeks later.
While there's no volume, it becomes a watched "do not worsen", not the test target.
2. Signal
Response rate, scheduling, useful click. It answers in hours or days.
It's what the loop optimizes at the start. In the LOOP-R example, the 48h response (20% → 30%) in place of conversion (3% → 5%).
3. Artifact
Does the output meet the rubric? Does it get the cases right? It runs in minutes, without waiting on anyone.
It's the ground of the lesson 2 experiment: a fixed set of cases and a command that gives the score.
Model judge
When the artifact score comes from a model with a rubric, calibrate first: it's course 1, lesson 6.
03The model and the rehearsal
The hypothesis is a text file with seven fields. hipotese.py checks it and, for an A/B test, does the sample math (two-proportion z-test, 95% confidence, 80% power).
File · kit/hipoteses/triagem.md
hipótese: incluir sinônimos de reembolso (estorno, devolução) nas regras leva o acerto de 0.70 a 0.80 métrica: acerto da triagem nos 40 casos de casos.jsonl atual: 0.70 alvo: 0.80 direção: maximizar medida por: python3 kit/medir.py <repo> sem piorar: acerto nos casos escondidos (fora do repo)
Terminal · cost-free rehearsal
K=~/projetos/loop-r-eng-aprende-v72/kit # git clone https://github.com/inematds/loop-r-eng-aprende-v72 ~/projetos/loop-r-eng-aprende-v72 for h in vaga triagem; do echo "## $h"; python3 $K/hipotese.py $K/hipoteses/$h.md; echo "saída: $?" done echo "## vendas: conversão 3% → 5%, 50 propostas por semana" python3 $K/hipotese.py $K/hipoteses/vendas-conversao.md --por-semana 50; echo "saída: $?" echo "## vendas: resposta em 48h 20% → 30% (métrica de sinal)" python3 $K/hipotese.py $K/hipoteses/vendas-resposta.md --por-semana 50 | tail -2
Example from the repository
## vaga HIPÓTESE: REPROVADA falta: atual, alvo, medida por, sem piorar hipótese vaga sem número saída: 1 ## triagem HIPÓTESE: OK — acerto da triagem nos 40 casos de casos.jsonl: 0.7 → 0.8 (maximizar) medida por: python3 kit/medir.py <repo> sem piorar: acerto nos casos escondidos (fora do repo) saída: 0 ## vendas: conversão 3% → 5%, 50 propostas por semana HIPÓTESE: OK — conversão (proposta que virou pacote pago em até 15 dias): 0.03 → 0.05 (maximizar) medida por: coluna resultado de dados/execucoes.csv, A contra B sem piorar: margem média, reclamações, opt-out amostra mínima: 1506 por variante (95%, poder 80%) — 3012 no total com 50 por semana: 61 semanas AVISO: mais de 12 semanas — teste uma métrica de sinal mais rápida e vigie esta saída: 0 ## vendas: resposta em 48h 20% → 30% (métrica de sinal) amostra mínima: 294 por variante (95%, poder 80%) — 588 no total com 50 por semana: 12 semanas
The loop-r/docs/05 rounds the same math to "~1,500 per variant, 60 weeks" (3% → 5%) and "~300, 12 weeks" (20% → 30%). With 50 proposals a week, conversion takes over a year; the response, three months.
Writing the target after seeing the result ("it rose to 26%, so the target was 25%"). The hypothesis goes in a file, with a commit, before the experiment; git shows the date.
And testing the money metric directly with low volume: the test never ends, and the temptation to stop "because it's winning" promotes noise (lesson 3).
04Practice
Rehearsal first, then your own hypothesis.
E2 · Your hypothesis
20 minPick one of your own processes that already has a number (an agent's hit rate, build time, tests that pass, replies). Write hipotese.md with the seven fields, with the measure as a command you can run today. If it's an A/B with people, run it with --por-semana and your real volume.
Done when: python3 $K/hipotese.py hipotese.md; echo "saída: $?" → HIPÓTESE: OK and saída: 0; and the "measured by" command, run now, prints the "current" value.
E3 · Signal instead of money
10 minIf your test went past 12 weeks, swap the target metric for a signal or artifact one, and move the money metric to "do not worsen".
Done when: python3 $K/hipotese.py hipotese.md --por-semana [your volume] → without the AVISO line.
05Lesson quiz
Question 1 of 4
"Improve the triage rules so support gets faster." What's missing to make it a testable hypothesis?skill
Correct
Without a starting number, an ending number, and a command that measures, any result will do. hipotese.py lists exactly those fields as missing.Not that one
Without a starting number, an ending number, and a command that measures, any result will do. hipotese.py lists exactly those fields as missing.Review: The model and the rehearsalQuestion 2 of 4
With 50 proposals per week, how long does it take to test a conversion going from 3% to 5%?skill
Correct
That's ~1,500 per variant, 3,012 in total: 61 weeks. That's why LOOP-R tests a signal metric and watches the money one.Not that one
That's ~1,500 per variant, 3,012 in total: 61 weeks. That's why LOOP-R tests a signal metric and watches the money one.Review: The model and the rehearsalQuestion 3 of 4
What is the role of a signal metric in the loop?
Correct
It responds fast and has a plausible link to the outcome. The signal → money link still has to be proven: in the LOOP-R example the reply rate went up and the conversion did not move significantly.Not that one
It responds fast and has a plausible link to the outcome. The signal → money link still has to be proven: in the LOOP-R example the reply rate went up and the conversion did not move significantly.Review: Three metric speedsQuestion 4 of 4
When should the hypothesis go into a committed file?
Correct
Written beforehand, it can't be adjusted to fit the result. Git keeps the date.Not that one
Written beforehand, it can't be adjusted to fit the result. Git keeps the date.Review: The model and the rehearsalQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Choose an answer. The explanation appears after you try.
06In three sentences
NumberA hypothesis has a metric, current, target, direction, measure command, and do-not-worsen; without that, it teaches nothing.
SpeedTest on the fastest metric that still links to the money one; watch the slow one.
Do the math firstThe minimum sample tells you whether the test fits your volume before you spend weeks.