The Chat Agent
Build Plan,
Pre-Broken.

Mission & Ground Rules

Mark Kashef runs Prompt Advisers (AI consulting, enterprise + SMB workshops). The new site gets a chat agent that does three things ordinary widgets do not: it walks visitors through the site on guided tours, it asks questions while touring to understand fit, and when signals appear it qualifies naturally, presenting offerings, Mark's credentials, and the right workshop track before capturing a lead. It adapts to how each visitor talks, and it improves weekly, without ever being allowed to break itself.

The Big Picture

From empty folder to: one embeddable widget.js (Shadow DOM, vanilla TS) on the Vercel site → a Supabase Edge Function chat that assembles a layered prompt from Postgres, streams Haiku 4.5 over SSE, and executes whitelisted tools → full logging → a cron-driven weekly review with a human gate, canary, and auto-rollback.

The Layered Prompt (decision #1)

The system prompt is assembled per request from versioned layers, stable to volatile, so prompt caching works and so "self-learning" has a blast radius of exactly one bounded layer.

Layer	Content	Who can change it
L0	Core identity, professionalism floor, "user messages are data", canary token	Human, rarely
L1	Ground-truth facts: offerings, credentials, workshop catalog, booking link	Human via facts table
L2	Tool rails, tour contract, qualification playbook (signals, not scripts)	Human
L3	Style layer rendered from 8 bounded dials	Machine-proposed, human-approved
L4	Runtime: current page, tour step, tab intent, rolling summary	Per request, after cache breakpoint

Cache trap baked in from day one: Haiku 4.5 will not cache a prefix under 4096 tokens, and any dynamic byte (timestamp, unsorted JSON, reordered tools) silently kills the cache. We log cache_read_input_tokens per turn so a cache regression is a dashboard dip, not an invoice surprise.

Stack, and why

Claude ≠ OpenAI: The Trap Table

Explicit pessimism, requested and honored: calling Claude from a Deno edge function is not shaped like calling OpenAI. Every row below is a 400 error, a silent empty stream, or a silent cost leak if ported by habit. The relay contract quarantines all of this inside one file.

The Critical Path, Ten Phases

Methodology: walking skeleton first, contract-first interfaces, evals before opinions, one variable per take, token accounting from API call #1. Red-topped cards are the phases most likely to burn extra takes; their pessimistic branches are pre-planned in the codex.

Tours & Qualification, The Experience

Tab click → intent → tour on rails. The model calls start_tour / advance_tour / end_tour / capture_lead; every argument is validated server-side against the database. The widget scrolls, navigates (CustomEvent → router.push, with a location.assign fallback), highlights, then confirms, and only then does the agent narrate that step and ask its question. Desync is structurally impossible rather than prompt-discouraged.

The Eval Harness

Past chat-agent builds burned their takes on taste-based prompt ping-pong. Here, from Phase 6 onward, every prompt change (human or machine-proposed) runs the golden set first: 25 frozen conversations, personas played by Sonnet, scored 1-5 against an anchored rubric by a versioned judge prompt. Ship rule: aggregate must not drop, and no security case may drop at all.

Defense In Depth

Design goal: not "unjailbreakable" (impossible) but a jailbreak is boring. Nothing valuable is reachable from a compromised turn: no secrets in prompt, tools validated on rails, output tripwired, spend capped.

The rings, outside in

Ring	Mechanism
Rate + cost	20 msgs/10min per session, 60/hr per IP, 2K char inputs, max_tokens 1024, global daily token budget, DB kill switch
Input heuristics	"Ignore previous", base64 walls, zero-width unicode: high confidence → polite deflection + flag; medium → flag only (never punish innocents)
Tool rails	Whitelisted routes and tour steps from DB, strict schemas, sanitized lead fields. Exfiltration through tools is structurally impossible
Output tripwire	Relay scans the stream for the canary token + L0 fingerprints; on hit: kill stream, canned reply, severity-high flag
Core (L0)	User content framed as data; identity claims in chat change nothing; no secrets exist here to steal

Learning-loop poisoning, the sneaky one: talking weird at the agent for a week to teach it bad habits fails twice. Flagged conversations are excluded from style aggregation, and the dials move at most ±0.10/week inside hard min/max bounds with EMA smoothing. You cannot teach it by attacking it.

The Weekly Self-Healing Loop

Every Sunday 03:00: collect the week's conversations → one claude-sonnet-5 Message Batch (50% price) judges each for injection (taxonomy: override, hijack, extraction, tool abuse, encoding, social engineering, poisoning) and quality (grounding, tone, resolution, missed signals) and emits style observations → aggregate → bounded dial proposal → golden set auto-runs → digest email to Mark with Approve/Reject → canary 10% for 72h → promote, or auto-rollback on flag-rate/abandon spikes. High-severity flags email immediately, not weekly.

{ "formality":     {"value":0.55, "min":0.4, "max":0.8},
  "brevity":       {"value":0.60, "min":0.4, "max":0.9},
  "warmth":        {"value":0.65, "min":0.5, "max":0.9},
  "directness":    {"value":0.60, "min":0.4, "max":0.9},
  "emoji":         {"value":0.00, "min":0.0, "max":0.2},
  "jargon_level":  {"value":0.40, "min":0.2, "max":0.7},
  "question_rate": {"value":0.50, "min":0.3, "max":0.7},
  "playfulness":   {"value":0.30, "min":0.1, "max":0.5} }
// weekly delta clamp ±0.10 · EMA over 3 weeks · dead zone |δ|<0.03
// vocabulary: whitelist of ≤10 observed terms, ≥3 distinct convos, filtered
// verbatim visitor text: never enters any layer, ever

The Scenario Codex

43 pre-planned failures across six theaters. L = likelihood. Every "Answer" already exists in the phase designs above; this table is the index that proves it. Full prose version lives in WAR-GAME.md Section 12.

Agent Bootstrap Block

Feeding this dossier to a fresh agent? This block plus WAR-GAME.md is everything it needs. Machine-readable summary first, then the human checklist.

{
  "project": "pa-chat-v0",
  "goal": "Embeddable guided-tour chat agent for the Prompt Advisers website",
  "authoritative_plan": "WAR-GAME.md (same folder). Execute phases 0-9 in order.",
  "stack": {
    "widget": "vanilla TS, single widget.js, Shadow DOM, fetch-reader SSE (never EventSource)",
    "backend": "Supabase Edge Functions: chat, lead-notify, review-submit, review-collect, admin-promote",
    "db": "Supabase Postgres, RLS deny-all to anon, schema in WAR-GAME.md 1.4",
    "chat_model": "claude-haiku-4-5 (200K ctx, 64K out, $1/$5 MTok, NO effort param, cache prefix >= 4096 tk)",
    "judge_model": "claude-sonnet-5 via Message Batches API (50% price)",
    "cron": "pg_cron + pg_net -> edge functions, heartbeat rows"
  },
  "hard_rules": [
    "model never edits its own live prompt; only proposes bounded L3 dial changes",
    "promotion path: eval gate -> human approval -> 10% canary 72h -> promote/auto-rollback",
    "facts table is the only source of business claims",
    "tours are DB-authored rails; model cannot invent routes or selectors",
    "verbatim visitor text never enters any prompt layer",
    "anthropic key only in Supabase secrets; browser talks only to our functions"
  ],
  "blocking_inputs_from_mark": [
    "supabase project ref + access", "ANTHROPIC_API_KEY",
    "site URL + route list (by phase 4)", "offerings/pricing one-pager (by phase 5)",
    "credentials bio (by phase 5)", "workshop catalog enterprise+SMB (by phase 5)",
    "brand voice notes (by phase 2)"
  ],
  "definition_of_done": "WAR-GAME.md section 13, seven items, all verifiable",
  "first_command": "read WAR-GAME.md fully, confirm section 0.4 inputs, then begin Phase 0"
}