A loop ends when the done criterion says it ended. If the criterion is vague, the loop never ends or ends too early; if it can be met by a shortcut, the agent finds the shortcut: it deletes the test that was failing, generates an empty page. The 0-to-4 scale of the execucao-longa kit names each step and shows the minimum accepted: level 3, a command with a lock.
3Locks that close the shortcut: count, frozen file, validator
4Function, regression and limit in one criterion
Goal: classify done criteria from 0 to 4 with the 5 questions and rewrite a vague criterion up to level 3, with a command and a lock.
01 · The done criterion ladder
Click each step: how it sounds, how the agent games it and what to do.
Lesson 601 · The done criterion ladderClick each step: how it sounds, how the agent games it and what to do.
1/6
0 · vague
There is no end you can check. The loop stops when the quota runs out or when the agent decides it's good enough. Example: "improve performance".
Scroll the page: each step below lights up its part of the diagram.
1/6
0 · vague
There is no end you can check. The loop stops when the quota runs out or when the agent decides it's good enough. Example: "improve performance".
2/6
1 · subjective
It looks like a criterion, but the agent's opinion decides: it approves itself. Example: "code reviewed and free of problems".
3/6
2 · gameable
It already has a command, but you can meet it without doing the work: deleting the failing test gives "0 failures"; twenty empty pages give "20 pages". It's the most dangerous step because it looks done.
4/6
3 · protected
A command plus a lock that closes the shortcut: a minimum count ("≥ 48 tests"), a frozen file or checked hash, a content validator. Example from the kit: "0 failures and ≥ 48 tests and tests/ untouched", not just "0 failures".
5/6
4 · independent
Level 3 plus someone or something outside checking: a real end-to-end test, a separate evaluator, a sample checked by a person. Use it when a mistake is costly. How to set up the separate evaluator is course 1.
6/6
Always climb to 3
Vague and subjective get rewritten; gameable gets a lock; protected is the minimum for leaving an agent alone; independent is for when a mistake is costly.
Scale from the goal.md template in the execucao-longa kit. Minimum accepted: level 3.
02 · The scale and the 5 questions, as they are in the template
Excerpt from execucao-longa/templates/goal.md. Paste it into your goal.md or the agent's rules.
templates/goal.md · scale and questions
SCALE (delete when filling in)
0 vague ......... "make it good", "improve" → rewrite
1 subjective .... "reviewed and clean" (the agent approves itself) → rewrite
2 gameable ..... "0 failures", "20 pages" (deletes a test, empty page) → add a lock
3 protected .... command + lock (minimum count, frozen file/hash, content validator) → minimum accepted
4 independent . level 3 + external check (real e2e, separate evaluator, human sample) → when a mistake is costly
5 QUESTIONS — each "no" drops the level
1. Can you check it with a command?
2. Is the answer yes/no (no "it improved")?
3. Is it impossible to meet without doing the work?
4. Does the proof show up in the output? (the /goal evaluator in Claude only reads the conversation)
5. Are function + regression + limit covered?
SIGNS OF A BAD CRITERION: "good/clean/adequate/complete"; depends on the agent saying it finished;
count only without checking content; doesn't say what must NOT change; verification takes hours.
Level 3 criterion · the three layers
**Function** — does what it should
- [ ] python3 -m pytest -q tests/ | tail -1 → 6 passed (and never fewer than 6)
**Regression** — didn't break what already worked
- [ ] python3 -m pytest -q | tail -1 → no "failed"
**Limit** — didn't touch where it shouldn't
- [ ] git diff --exit-code -- tests/; echo $? → 0
- [ ] sha256sum -c tests.sha256 → tests/test_conta.py: OK
03 · From 0 to 3, one step at a time
The same request rewritten until it's shortcut-proof.
1. 0 → 1
"Improve the word-counting function" becomes "the function counts words correctly".
It's still opinion: who says "correctly"?
2. 1 → 2
"python3 -m pytest -q prints 0 failed".
Now you can check it. But deleting the tests also prints 0 failed.
3. 2 → 3
"pytest prints 6 passed, tests/ did not change (git diff empty) and the test file hash matches".
Minimum count and a frozen file: there is no way to meet it without doing the work.
Add a check the agent doesn't control: another agent that only evaluates, an end-to-end test in the real environment, a sample you check yourself.
This step is the subject of course 1.
04 · Practice
On your machine. Exercise 2 uses a project of yours that has tests.
E1 · Classify and rewrite
15 min
In criterios.md, write three criteria you have already used (or seen) to say that a task was finished. Next to each one, put the level from 0 to 4 and, on a line that starts with nível 3, the rewritten version. Write the marker nível 3 exactly like this, in Portuguese: it is what the grep counts.
Done when:grep -c 'nível 3' criterios.md → 3, and grep -ciE '\b(good|clean|adequate|complete|proper)\b' criterios.md only finds words in the lines of the original criteria, never in the rewritten ones.
E2 · A level 3 criterion that runs
15 min
In a project of yours that has tests, run the full criterion: the suite with a minimum count and the lock that the tests did not change.
Done when:python3 -m pytest -q | tail -1 (or your project's test command) shows the number of tests, with no failed, and git diff --exit-code -- tests/; echo $? → 0.
E3 · Prove the lock catches it
10 min
In the same folder, simulate the shortcut: replace the content of a test with an empty test, run the locks, and then undo it with git checkout.
Done when: With the test replaced, git diff --exit-code -- tests/; echo $? → 1; after git checkout -- tests/, the same command → 0.
05 · Lesson quiz
Question 1 of 4
"python3 -m pytest -q prints 0 failed". What level is it?competency
Correct
It has a command, but deleting the tests also gives 0 failed. The lock is missing: minimum count and untouched tests. The common mistake is thinking a command is already level 3.
Not that one
It has a command, but deleting the tests also gives 0 failed. The lock is missing: minimum count and untouched tests. The common mistake is thinking a command is already level 3.Review: The done criterion ladder
Question 2 of 4
Which of the 5 questions knocks down "the agent confirms it reviewed everything"?
Correct
There is no command, the answer depends on the agent saying it finished (a sign of a bad criterion), and it can say so without having done it. Level 1.
A criterion covers function and regression, but says nothing about files that could not change. What is missing?competency
Correct
Function, regression and limit: without a limit, the agent can meet the rest by touching where it shouldn't. "Doesn't say what must NOT change" is among the signs of a bad criterion.
