💸 Cost: More Than 1 Million Tokens
The most tangible problem with loop engineering is the cost. Each orchestrator round consumes reasoning tokens to plan, each worker consumes tokens to execute, and context passing between components multiplies everything. Cole Medin demonstrated this with real numbers on his dashboard.
"In a single run, here are all the loops the orchestrator went through; I spent over a million tokens just to build a relatively simple app."
📊 Why It’s So Expensive
Orchestrator reasoning
The orchestrator needs to read the entire spec, plan the round, create prompts for each worker, and then evaluate the results. Each of these operations consumes input and output tokens.
Distributed context passing
Worker results need to get back to the orchestrator. The state store needs to be read at every round. Information flows in every direction, and each flow costs tokens.
Single model for everything
When you use only /loop in Claude Code, the same expensive model (Opus/Sonnet) is used for everything: triage, planning, execution, validation. Without a model mix, the cost per token is as high as possible at every stage.
"One of the other big issues I see with using slash goal or routines or loops in Claude Code is you're just using one model for pretty much everything. That's part of the problem of why it's so expensive."
📉 Quality: "Run for a Day, Come Back to Crap"
The second major problem is the quality degradation in long loops without human supervision. Cole Medin experienced this firsthand and describes the situation plainly: you set up the loop, walk away, and come back to find the output is unusable.
"A lot of times, people set these systems up to just run, run, run, run. Then you let them run for a day. When you come back, all you find is garbage. That’s happened to me while testing a bunch of things in Claude Code, like routines and /loop."
🕵 Why Quality Degrades
- ✗No human validation: the agent validates itself, but LLMs can accept mediocre results as "correct"
- ✗Accumulation of bad decisions: each round builds on the previous one; if round 3 accepts a bug, rounds 4-10 build on that bug
- ✗Context fatigue: even with external state, the agent can lose sight of the big picture of what it should build
- ✗Local optimization: each round optimizes for its task, but may hurt the coherence of the system as a whole
🛡 The Safeguard: Human-in-the-Loop
Cole emphasizes that the solution isn’t to abandon loops, but to add checkpoints where a human validates the work before the next round: "We can always have it pause for us to validate something before it continues, which is one of the biggest problems with loop engineering right now in general." The rule is clear: the agent does the heavy lifting, and the human ensures quality.
💥 Context Bloat: The Overloaded Session
The third problem Cole identifies is specific to the basic use of /loop in Claude Code: context bloat. When the loop runs in the same agent session, each iteration adds more history to the context window. After many iterations, the LLM becomes literally overloaded.
"In a lot of setups, you don’t actually work across different coding agent sessions. When you use /loop in Claude Code, for example, it just keeps going in the same agent session. So if you leave it looping for a while, your LLM’s context gets huge, and it gets overloaded."
🧠 What Happens to the Context Window
Iteration 1–5
HealthyThe context window has plenty of room. The agent has access to the original instructions, spec, and recent history. High quality.
Iteration 10–20
PressuredThe context window starts to fill up. Initial instructions get pushed out of the attention window. The agent starts to “forget” details.
Iteration 30+
OverloadedContext window overflow. The agent generates repeated code, ignores constraints, and loses coherence. The result: expensive garbage. This is where “run for a day, come back to crap” happens.
🔧 The Solution: Distribute Across Sessions
Cole proposes the solution: "We need a system where we can actually distribute the work among different coding agent sessions and make it so they can all communicate with each other." Instead of one long session, use multiple short, focused sessions coordinated by external state. Each session has fresh context and focuses on the task at hand.
🏗 Complexity: Heavy Engineering
Building a robust loop engineering system is not trivial. Requires significant engineering: state store, worktree management, cost tracking, observability, human-in-the-loop, retry logic, orphaned worker cleanup, port conflict resolution, database branching. That's a lot of infrastructure to support what Cole calls something that "doesn't deserve its own buzzword."
🧰 What You Need to Build
State Store
Postgres/Neon to persist state between rounds. Schema for runs, events, tasks, cost.
Observability
Dashboard to view orchestrator decisions, worker progress, and accumulated cost in real time.
Worker Isolation
Worktree management, database branches, port conflict resolution for parallel workers.
Control Plane
Pause, resume, and cancel mechanisms. Human-in-the-loop checkpoints. Automatic budget limits.
"There's a lot of engineering that goes on behind the scenes here."
🌀 Compounding Hallucinations Across Rounds
Perhaps the most insidious risk of loop engineering is the compounded hallucination: when an LLM error in one round goes undetected and carries over to subsequent rounds, with each iteration amplifying the problem. This differs from a one-off error because it builds layers of incorrect code that look correct on the surface.
⚠ The Snowball Effect
Round 1: The worker implements a function with a subtle bug in edge case handling. The orchestrator validates and accepts it (the LLM doesn't detect the bug).
Round 3: Another worker builds a feature that depends on the buggy function. The code "works" on the happy paths, but inherits the bug.
Round 7: The entire codebase is built on the assumption that the round 1 function is correct. Fixing it now requires refactoring everything. The cost of fixing it exceeds the cost of doing it manually.
🛡 Defenses Against Compounding Hallucinations
- ✓Automated tests as a validation gate between rounds: if the tests fail, the round is re-executed
- ✓Human-in-the-loop at critical points: human review before rounds that build on previous work
- ✓Round rollback: the ability to undo an entire round when validation fails
- ✓Cross-validation: use a different model to validate the main worker's output
❓ Questions
The disadvantages aren’t a reason to dismiss loop engineering, but they are a reason for healthy skepticism. Cole Medin raises the most direct questions.
Is loop engineering really the best approach?
Cole is direct: "There is no way you're gonna convince me that loop engineering is the way to get the best results possible with AI coding assistance." The claim is strong and intentional. Loops optimize for throughput, not quality. For the best results, a human in the loop is irreplaceable.
Boris manages tens of thousands of agents per day. Does that make sense?
Cole directly questions how practical this is: "Boris Cherny says that there are days he manages tens of thousands of AI agents at once. Like really, is that actually practical? Is that going to scale?" Then he jokes: "I mean, maybe that does explain some of the bugs we have in Claude Code."
Does the budget privilege distort your perspective?
The biggest advocates of loops operate with resources most people don't have. Boris has Anthropic. Peter has "pretty much an infinite budget." When people promoting a practice don't feel the cost, it's easy to be enthusiastic. The question is whether this works for people who count every token dollar.
🤔 Reflection Point
Cole ends his analysis of the drawbacks with a constructive proposal: don’t discard loops, but wrap them in a harness that addresses the problems. "I like the concept of it, and I want to drive how autonomous my coding agents can be, but you got to have the right system." The next module (2.3) will cover exactly when to use loops and when not to.
📋 Module Summary
Next Module:
2.3 — When to Use and When Not to Use