🧪 The test: 38 tasks, 4 waves, graded by code
Before the numbers, a benchmark. The field test (2026-06-22) put 38 tasks divided into 4 waves of increasing difficulty, adding up 334 graded calls. "Graded by code" means that each answer was evaluated using hidden tests — the key was verified by an independent solver before any paid call. No human judge, no guesswork: it either passed the test or it didn't.
The prompt was identical on both sides (Fugu and Opus), with 3 to 7 runs per task to measure consistency, and partial credit For multi-rule tasks (you gain in proportion to the number of rules followed). This is the foundation that gives weight to everything that comes after.
🆕 New here? Three terms
- Code-graded: the score comes from automated tests, not opinion — removing evaluator bias.
- Hidden tests: the evaluation cases aren’t shown to the model, so it can’t “memorize” the answer.
- Partial credit: on tasks with multiple rules, getting 4 out of 5 right is 4/5, not zero.
🌊 The 4 waves
| Wave | Tasks | Character |
|---|---|---|
| Wave 1 | 14 | “textbook-hard” puzzles (the baseline ceiling is already high) |
| Wave 2 | 11 | “designed traps” (designed traps) |
| Wave 3 | 6 | “dense multi-rule specs” (dense, multi-rule specs) |
| Wave 4 | 7 | “failure frontier” (designed to break 1 model) |
Total: 14 + 11 + 6 + 7 = 38 tasks. Each wave is harder than the last—the Wave 4 was designed to find where a solo model gets stuck.
🎯 Who was more accurate
The accuracy score across all 38 tasks: 36 ties · 2 soloist wins (Opus) · 0 team wins (Fugu). Reading wave by wave, the hidden team never found an answer the soloist didn’t. In the words of the field test: “not once in 38.”
🏁 Accuracy by wave (Fugu / Opus / tie)
| Wave | Fugu Wins | Opus Wins | Ties |
|---|---|---|---|
| Wave 1 | 0 | 1 | 13 |
| Wave 2 | 0 | 0 | 11 |
| Wave 3 | 0 | 1 | 5 |
| Wave 4 | 0 | 0 | 7 |
| Total | 0 | 2 | 36 |
Opus’s 2 wins came in Wave 1 and Wave 3 (detailed in topic 5). The team didn’t win any, in any wave.
⚠️ The raw reading
More models did not produce more accuracy. The promise: "combine several and exceed the solo ceiling" (Path 1) Wasn’t confirmed In this benchmark, the solo ceiling was already high enough to match almost everything, and the team’s marginal gain was exactly zero in accuracy.
⏱️ The clock: 4.5× slower
If accuracy was a tie, the clock wasn’t. Across the entire exam, Fugu took 357 minutes against 80 minutes of Opus — about 4.5× slower. For simple code, the contrast is stark: Opus responded in 3–6 seconds, while Fugu took 2–7 minutes — for the same answer.
Average latency per task, Fugu (blue) vs. Opus (cyan). In every wave, the team is multiple times slower—the cost of waiting for the entire fan-out.
📊 Average latency per task (seconds)
| Wave | Fugu | Opus | Approx. ratio |
|---|---|---|---|
| Wave 1 | 205s | 40s | ~5× |
| Wave 2 | 110s | 19s | ~6× |
| Wave 3 | 161s | 6s | ~27× |
| Wave 4 | 97s | 15s | ~6× |
| Total | 357 min | 80 min | 4.5× |
Wave 3 (dense specs) was the most disproportionate: Opus solved in 6s what took Fugu ~161s — same answer, ~27× the time.
💸 The bill: 5× more expensive
The final exam math: US$ 53,60 (Fugu, 163 calls) against US$ 10,66 (Opus, 171 calls) — about 5× more expensive. Notice: Fugu even did less external calls, but it cost much more. The reason is the hidden cost of Track 1, now itemized: most of Fugu's spending goes to orchestration tokens — what the private team reads and writes internally, all billed.
Average cost per task, Fugu (blue) vs. Opus (cyan). Wave 3 is the most uneven: ~$0.358 vs. ~$0.014—the team costs ~26× as much to tie.
📊 Average cost per task (US$)
| Wave | Fugu | Opus | Approx. ratio |
|---|---|---|---|
| Wave 1 | $0.389 | $0.100 | ~4× |
| Wave 2 | $0.297 | $0.046 | ~6× |
| Wave 3 | $0.358 | $0.014 | ~26× |
| Wave 4 | $0.276 | $0.037 | ~7× |
| Total | $53.60 (163 calls) | $10.66 (171 calls) | 5× |
More expensive with less calls: the cost isn't in the number of calls, it's in the team's internal tokens. You pay for the entire pool, not just the final response.
🥈 The team's 2 losses
Of the 38 tasks, only 2 didn't tie—and in both cases it was the solo that it won. The team didn’t just fail to win anything: where there was a difference, it lost. Coordination doesn’t prevent slipups; sometimes it adds points of failure.
Wave 3 — "fix every bug in stats module"
The module had 6 bugs. Opus caught all 6 on every run (100%). Fugu repeatedly missed 1 — it ended up at 80% (5/6). Same task, and the team consistently let a bug slip through.
Wave 1 — "strict Roman numeral validation"
The rule was to reject invalid forms like “IIII”. Opus did 5/5; Fugu did 4/5 — slipped once, accepting what it should have rejected.
📊 The 2 losses, side by side
| Task | Fugu | Opus | Winner |
|---|---|---|---|
| Stats: 6 bugs | 80% (5/6) | 100% (6/6) | Opus |
| Strict Romano | 4/5 | 5/5 | Opus |
⚠️ The lesson
When the benefit of combining models is zero (correctness) and the team still slips where a solo model gets it right, the cost-benefit calculation strongly favors the single model—exactly what the ladder in the next topic formalizes.
🪜 The decision ladder + setup in Claude Code
The verdict isn’t “orchestrators are bad” — it’s “use the right criteria.” The field test proposes a decision ladder: a ladder of 3 questions that tells you, for each task, whether the orchestrator earns its place or whether a single strong model is the best buy.
The ladder: the orchestrator wins only on the final rung. In this test, every task "fell off the top" — a strong model already tied, faster and cheaper.
🪜 The 3 steps
- 1.Can a strong model already solve it? → YES: use 1 model (faster, cheaper, and a tie).
- 2.NO → you want 1 endpoint + a vendor hedge, e the wait and the bill don't matter? → NO: 1 model is still the best buy.
- 3.YES → that’s when the orchestrator earns its place.
✓ When to choose Fugu
- ✓Want zero orchestration code + 1 endpoint
- ✓Want a hedge against vendor lock-in
- ✓Cost and wait time don't matter in your case
✗ When to stick with 1 strong model
- ✗A top model already solves the task
- ✗You care about speed (the team adds minutes)
- ✗You care about cost (you pay for the hidden team)
🛠️ Hands-on: conceptual Fugu setup in Claude Code
At the conceptual level, the idea is to point Claude Code to the Fugu endpoint using your API key—the same “base URL + API key” pattern you use with any compatible provider. Honesty first: the steps exact (correct endpoint, variable names, billing) come from Sakana's official documentation (sakana.ai) and the author's setup markdown. The field test reports that Fugu's setup Isn’t trivial — it’s not just “swap the endpoint + key” like with a typical OpenAI-compatible provider. Treat the example below as a skeleton, not a literal recipe.
# Esqueleto conceitual — substitua pelos valores reais da doc da Sakana. # NÃO é um endpoint garantido; confirme em sakana.ai e no .md do autor. export ANTHROPIC_BASE_URL="<base-url-da-sakana>" # ex.: a URL do gateway Fugu export ANTHROPIC_API_KEY="<sua-api-key>" # sua chave do Fugu # depois, no Claude Code, você apontaria a sessão para esse provider # e usaria o modelo "fugu-ultra" conforme a doc indicar. claude --model fugu-ultra "resuma este repositório e aponte 3 riscos"
Billing: there are two ways — subscription (monthly plan, like the author who exceeded the 5h window on the US$ 200/month plan) or pay-as-you-go via API (pay per token used). To experiment without committing to a plan, pay-as-you-go is usually the way to go—but remember the hidden cost: you pay for the orchestration tokens of the team. Confirm values and endpoints in sakana.ai before running.
Quick recall (optional): in the field test, what was the headline result?
✅ Module summary
Next track:
Track 3 — OpenRouter Fusion: the ensemble (same prompt, N models, one judge)