♻️ Self-improving systems

There’s a tempting myth: “if I use the smartest (and most expensive) model, it’ll find those deep bugs that no other model can find.” In the conversation, it was said that one model found deep bugs and failures of security that other models hadn’t seen. The easy reaction is: “See? You need the top model.” Pocock disagrees at the root. In his words: "you could find these bugs with cheaper models if you looked in the right places and gave it the right prompt/harness". In other words, what the “other models” lacked wasn’t intelligence — it was direction.

This connects directly to what you saw in Track 1 (Token Economy): a harness well, you let a “dumber” model do the same work. Here, the idea is the same, applied to bug discovery. Instead of paying a lot for a genius model to run once, you build a system that puts a simple model in the right place, repeatedly. The reason matters: an expensive model is a cost that grows with every call; a well-targeted system is an asset that pays off every day. The common mistake is confusing “it found a hard bug” with “I always need the harder model”—when what actually found the bug was where e how you looked.

Pocock’s concrete recipe: run a cron journal of security review that checks, every day, a new part of the repository, using a relatively simple model. The keyword is rotating: you don’t try to scan the entire project at once (expensive, shallow, and blowing up the context window). You break it into pieces. Today, the authentication folder; tomorrow, payments; then uploads. In a few weeks, the entire repo has been audited—and the cycle starts again, catching what’s changed.

Why scheduled and not “when I remember”? Because the lesson Pocock draws from an incident is direct: "what did you learn? That there are security issues in your code — you should have something that runs and checks for more in the future." Security can’t depend on your memory on a busy day. The reason of the simple model: a sliced review is a small, well-scoped task—exactly where a cheap model is enough (topic 1). The common mistake is setting up a cron job that reviews the entire repo at once: it gets expensive, creates noise, and nobody reads the huge report. Small, daily, rotating, actionable.

Below, the copyable artifact: a line of crontab that runs a headless security review agent every day at 6 a.m., pointed at “the day’s slice.” Add it to your crontab (crontab -e) and adjust the repo path:

# Edite com:  crontab -e
# Roda TODO DIA às 06:00 — checa uma fatia rotativa do repo (dia-do-mês % nº de pastas).
0 6 * * * cd /caminho/do/repo && DIRS=(auth payments uploads api db) && TARGET=${DIRS[$(( $(date +\%d) \% ${#DIRS[@]} ))]} && claude -p "Faça um SECURITY REVIEW so da pasta ./$TARGET. Liste cada falha (arquivo:linha), severidade (alta/media/baixa) e a correcao sugerida. Se nada grave, responda 'OK'. Saida em markdown." --model claude-haiku-4-5 --allowedTools "Read,Grep,Glob" >> ~/security-reviews/$(date +\%F)-$TARGET.md 2>&1

Here, self-improvement stops being proactive (the cron) and becomes reactive, but automated. Pocock’s example: a bug in telemetry appears in a tool like the Sentry. Instead of reading, copying, and manually opening an issue, the system automatically creates an issue and the brand with a label (label) like agent:explore. That label triggers an agent—the exact queue pattern from module 4.4.

The agent then returns structured data: can it be fixed now, or does it need a human? If it’s straightforward, it implements the fix and opens the PR, and the flow moves on to review—eventually with a auto-merge (runs on its own) or a ping for you to decide. Pocock puts it all together in this phrase: "push the human-in-the-loop checkpoints as close as possible to the final result." In other words: the machine handles the whole process; you step in near the finish line, where your attention goes further. The common mistake is letting the pipeline auto-merge everything — with no human gate on changes that affect behavior, you trade manual work for silent risk.

This is the module’s turning point, and Pocock sums it up with an image that sticks: "if someone keeps stealing your bike, maybe get a lock." Fixing the bug that appeared is treating the symptom. The question that truly improves the system is about the root cause (root cause): why did this bug exist for so long without you noticing? What was missing from your process — a test, a review, a lint check — that let this slip through until it became an incident?

Notice the difference in level. "I found a security bug and fixed it" stops at the bike. "There are security flaws in my code, so I'm going to put together something that runs and checks for more in the future" installs the lock—it’s literally Topic 2’s cron job born from an incident. Each bug becomes more than just a fix; it becomes a question: Which system guardrail let this through? The answer becomes a test suite, automated review, or refactor. The common mistake is stopping happily at the fix: you close the issue, feel like you solved it — and the same kind of bug comes back through another door, because the lock was never bought.

Putting it all together: a self-improving system is made up of several loops that reinforce each other. Pocock names three concrete ingredients: test suites (that catch regressions before they reach the user), human reviews (that provide insight into the system, not just the code) and refactor (which reduces the chance of the next bug appearing). Each incident feeds these loops: it becomes a test, a review checklist, or a cleaner part of the code.

The critical detail—and this is where it connects to module 4.4 (Loops × Queues)—is that “self-improving” no means "leave an infinite loop running on its own and disappear." It means setting up triggers (cron, telemetry, label) that put tasks in a queue, and feedback loops that make each pass better than the last. You’re still in charge: you prioritize, review the risky parts, and adjust the triggers. The common mistake is imagining a 100% autonomous system that fixes itself forever without supervision—that doesn’t exist; what does exist is a system that does the repetitive work and calls you in when it matters.

The thesis that closes the module is Pocock’s most important statement here: "review not just the code — also review the SYSTEM that produces the code." In the agentic era, the agent writes most of the code. So your review focus moves up a level: less "is this line correct?" and more "the process that generated this line trustworthy?” The prompts, skills, crons, review gates — this is the plumbing that, when tuned, keeps the next bug from ever being born.

This also answers a dangerous question that comes up in the next module (4.6): "who reviews the AI that says everything is okay?" Response: you, reviewing the system. That’s why it’s worth checking some PRs from time to time that the agent assured you are okay—not out of paranoid distrust, but to improve the system over time. Below is a practical summary in the form of a copyable checklist — run it whenever a bug slips through, to turn the fix into a system improvement:

Bug escapou? Antes de fechar a issue, COMPRE O CADEADO:
[ ] CAUSA RAIZ — por que esse bug existiu tanto tempo sem ninguém notar?
[ ] TESTE — adicionei um teste que pegaria esse caso na próxima vez?
[ ] DETECÇÃO — meu cron de security/review olharia essa parte? Se não, ajusto o rodízio.
[ ] PROCESSO — que gate (review, lint, label) deixou isso passar? Reforço ele.
[ ] SISTEMA — revisei o código OU o sistema que produz o código? (era pra ser os dois)
Se você só corrigiu o bug, comprou outra bike. Instale o cadeado.