PTENES
MODULE 4.3

πŸ€– Advanced Loop: Multi-Agent Orchestrator

The complete system: spec document as input, orchestrator plans rounds, workers execute, state store persists, human-in-the-loop validates.

6
Topics
30
Minutes
Adv.
Level
Hands-on
Type
1

πŸ“„ Spec Document as Input

At the advanced level, the input is no longer a plan.md with 5 tasks. It's a complete spec document that describes an entire application. The orchestrator reads the spec, divides it into rounds, and dispatches workers to build everything.

"We send in our initial spec to the orchestrator and it has to reason about that and then figure out how many workers to spin off."
-- Cole Medin

πŸ“‹ Spec Example (Kanban Board)

# Spec: Single Page Kanban Board

## Objetivo
Criar um Kanban board como web app estatico (HTML + CSS + JS).

## Requisitos
- 3 colunas: To Do, In Progress, Done
- Drag and drop entre colunas
- Persistencia via localStorage
- Responsivo (mobile-first)
- Tema dark com cores amber/neutral

## Criterios de Aceitacao
- App carrega sem erros no browser
- Drag and drop funcional
- Dados persistem apos reload
- Layout correto em mobile (320px+)

Cole Medin demonstrated exactly this scenario: "building a single page Kanban board as a static web app, I just take this prompt and I'll show you it running live right now." The orchestrator received the spec and automatically split the work into rounds.

2

🎯 The Orchestrator Plans Rounds

The orchestrator analyzes the spec and decides on the strategy: how many rounds, how many workers per round, and which tasks each worker gets. This is the system's most critical decision, which is why it requires a strong model.

"The orchestrator is deciding how to split up the work right now. The orchestrator spent 6,000 tokens with that initial planning and then prompting our first three workers in round number one."
-- Cole Medin

πŸ“‹ Multi-Agent Orchestrator Prompt

You are the orchestrator for a multi-agent build system.

Input: spec.md (attached)
State store: Postgres (Neon) at DATABASE_URL

Your job:
1. Read spec.md and break it into independent work units
2. Plan rounds: group units that can run in parallel
3. For each round, dispatch workers with specific prompts
4. After each round, read worker results from state store
5. Validate: does the output match spec requirements?
6. If validation passes, plan next round. If fails, retry or escalate.
7. When all spec requirements are met, report completion.

Pause for human approval before starting each new round.
spec.md input Orchestrator plans rounds validate results Round 1 3 workers Validate HITL Approve Round 2 2 workers Validate HITL Approve Round N Done State Store (Postgres/Neon)
3

βš™ Workers Execute and Report

Each worker receives a specific task from the orchestrator and operates in own session with an isolated worktree and database branch. On completion, it writes the results to the state store (Postgres), not the context window.

"Those workers are going to go back and they're going to update the state that we have in our database. And so this is our loop, right? Because then the next time the orchestrator runs, it's going to get that updated state from the workers."
-- Cole Medin

πŸ“‹ Prompt Template for a Worker

You are Worker {{worker_id}} in Round {{round_number}}.

Your task: {{task_description}}
Spec context: {{relevant_spec_section}}
Previous round results: {{previous_results_summary}}

Instructions:
1. Execute the task in your worktree
2. Run validation: {{validation_command}}
3. Report results to state store:
   - Status: success/failure
   - Files changed: list
   - Test results: pass/fail count
   - Notes: any issues encountered

Do NOT modify files outside your assigned scope.
4

βœ… Orchestrator Validates and Launches the Next Round

After each round, the orchestrator reads the results from the state store, checks whether the spec criteria have been met, and decides whether to launch another round or declare completion. This is the checkpoint that distinguishes a robust system from a blind loop.

"It’ll move forward one round at a time, validating each one. We can also have a person review what happened in the first round before the orchestrator moves on to the next."
-- Cole Medin

πŸ”§ Validation Logic Between Rounds

After Round {{N}} completes:

1. Query state store for all worker results in this round
2. Check: did all workers report success?
   - If any failed: retry failed workers (max 2 retries)
   - If still failing: pause and escalate to human
3. Check: do combined results satisfy spec requirements?
   - Run integration tests across merged worktrees
   - Compare output against spec acceptance criteria
4. If all good: plan Round {{N+1}} with remaining work
5. If spec is fully satisfied: merge all branches, report done
5

πŸ›‘ Human-in-the-Loop Checkpoints

Human-in-the-loop (HITL) is what separates a reliable system from an experiment. Without HITL, you risk letting the loop run for hours and coming back to find that the result is unusable.

"A lot of times, people set these systems up to just run, run, run, run. Then you let them run for a day. When you come back, all you find is garbage."
-- Cole Medin

πŸ“‹ Where to Place Checkpoints

1

After planning

Review the rounds planned by the orchestrator before it starts dispatching workers.

2

Between rounds

Validate the previous round's results before authorizing the next one.

3

Before the final merge

Review all changes before merging into the main branch.

🎯 Reflection Point

Cole Medin sums up the philosophy: "That is the kind of reliability that I feel like we really need to have right now in order to build anything more than simple demos." HITL isn’t optional in production systems. It’s what turns a demo into something usable.

6

❓ Questions

The multi-agent orchestrator is the peak of complexity. Questions that come up in real-world use.

Is the cost justified?

Cole reports using more than 1M tokens for a single simple application. Multi-agent multiplies that. The honest answer: it depends on the value of the output versus the cost of a human developer. It may be worthwhile for PoCs. For production, calculate the cost first.

When complexity becomes the enemy

Not every project needs a multi-agent orchestrator. Cole is clear: "I would just fold loop engineering into harness engineering." If a simple plan.md does the job, don't scale up to multi-agent just because it seems more sophisticated.

Orchestrator model vs. worker model

Cole uses Pi/Kimi for the orchestrator (routing decisions) and Claude for workers (actual implementation). The orchestrator doesn't need the most expensive model, but it does need reliable reasoning. Test before deciding.

🎯 Reflection Point

Multi-agent is a powerful tool, but it isn't a universal solution. Use it when the problem truly requires parallelism, rounds, and validation between stages. For most scenarios, the issue bot (4.2) or even the task runner (4.1) is enough.

πŸ“‹ Module Summary

βœ“
Spec document -- complete input describing what to build
βœ“
Orchestrator plans rounds -- divide work into parallelizable waves
βœ“
Workers + state store -- run in isolated sessions, report via database
βœ“
Validation between rounds -- orchestrator checks before moving forward
βœ“
HITL checkpoints -- human approves at critical points

Next Module:

4.4 -- Anti-Patterns and Troubleshooting (what goes wrong and how to fix it)