📊 The A/B/C ablation plan

Ablation is a term borrowed from research: removing part of a system to measure what it actually contributed. Here, the removed part is instructions from your config. And measurement requires at least two configs running the same thing—in practice, three: A, B e C.

A is your current config, intact—the baseline (the honest point of comparison, what you have today and must not let get worse without noticing). B is the simplified version, with about 50% fewer instructions: you kept what seemed essential and cut the rest. C is the minimal version: essential context + objective + guardrails + quality criteria + verification. Nothing more.

What to look at: the columns shrink from left to right, but the cyan bands run across all three with the same length. That detail is what makes the test meaningful: only one variable changes (the config). If you also change the task, model, or input prompt, the result no longer tells you anything about the cut.

A real task is one you’ve already done or will do anyway this week: publish a post, refactor a module, generate a report, build a page. A hypothetical task—“ask the model to write a Fibonacci function”—doesn’t test any of your config, because your config wasn’t written for Fibonacci.

Five tasks is the minimum to keep the result from being a matter of luck; ten is the practical ceiling before the round becomes a project in itself. The set needs to be representative: if 70% of your work is writing content and 30% is working with code, the grid should reflect roughly that proportion.

"It got better" isn't a measure. Nine dimensions cover what matters, and each has a simple way to score it: a scale of 1 to 5 when judgment is unavoidable, objective counts whenever possible. Counts are always preferable — "3 fixes" leaves no room for debate; "quality 4" does.

Three of these dimensions are often ignored, and they’re exactly the ones that reveal bloated config: autonomy (how far the model gets on its own before stopping to ask), adherence (whether it followed the rules you wanted keep — not all the rules that existed) and the ability to verify its own work.

Three things stay frozen in every run: same task, same model, same input prompt. Only the config changes. Comparing yesterday’s A, on the old model, with today’s C on a similar task isn’t ablation — it’s anecdotal.

And there’s a rule that separates those who measure from those who convince themselves: write down what "good" would look like before looking at the answer. If you define the criteria after reading the output, your brain adjusts the criteria to the output — you’ll think what came back is great and won’t notice that you moved the goalposts. Write down the criteria first, then run it.

# Grade de ablação A/B/C

Modelo usado: <nome do modelo>      Data: <aaaa-mm-dd>
Regras-alvo (para aderência): <liste as N regras que você quer manter>

## Tarefa 1 — <sua tarefa aqui>
Prompt de entrada (idêntico nas 3): <cole o prompt>
O que seria BOM (escrito ANTES de rodar): <seu critério aqui>

| Versão | Qualid. 1-5 | Aderência n/N | Autonomia (interrupções) | Correções | Consistência | Tempo (min) | Ferram. inúteis | Complexidade (linhas) | Autoverificação |
|---|---|---|---|---|---|---|---|---|---|
| A (atual)      |   |   |   |   |   |   |   |   |   |
| B (50% menos)  |   |   |   |   |   |   |   |   |   |
| C (mínima)     |   |   |   |   |   |   |   |   |   |

Observações (o que quebrou, em qual versão, com trecho):
- <anote aqui>

## Tarefa 2 — <sua tarefa aqui>
(repita o bloco acima)

Reading the grid has three outcomes, and only three. Where C tied with A, the instructions that were in A and disappeared in C were dead weight: they stay out, permanently. Where C performed worse once, that doesn’t count—a bad run is noise, and restoring an instruction because of it is like rewriting the entire config every time the model yawns.

Where C performed worse repeatedly, that’s when the reintroduction rule comes in. And there’s a fourth case that comes up often and surprises people: B beating A and C. When this happens, you’ve found the sweet spot—that’s usually where the config stays.

What to look at: there are four possible outcomes, and only a returns instructions for the config. Notice that “C got worse” alone isn’t an outcome—it has to pass through the second diamond. This flowchart exists to help you resist the urge to rewrite the config at the first frustration.

One eval is a test case for a model: a concrete task with a success criterion you can check. Every failure that showed up in your grid is already a ready-made eval— just write "this task, this criterion, the model failed here" and save it.

Evals age too: they typically last 1 to 3 model generations. When an eval passes every time, in every version, it got saturated — no longer separates anything and only costs execution time. Retire it and create new ones where you see the current model stumble.

How to read this example table: it’s illustrative—the numbers are made up. What matters is the "Reading" column: four tasks, four different outcomes. A real grid rarely points to a single verdict, and that’s exactly why it’s worth more than your intuition.

Rodada de ablação — versão <A | B | C>

Tarefa: <sua tarefa aqui, exatamente como você a pediria num dia normal>
Entradas: <arquivos, links, dados reais que a tarefa usa>

Faça a tarefa até o fim. Ao terminar, responda também:
1. Quais critérios você usou para decidir que estava pronto?
2. Como você verificou o próprio trabalho? Cite a evidência (comando,
   arquivo conferido, teste rodado).
3. Em que ponto você ficou em dúvida e teria perguntado a um humano?

Não peça confirmação no meio do caminho a menos que seja bloqueante.