Define observable success
What is it: Each task specifies its initial state, expected result, prohibited effects, and tolerances.
Why learn this: Without an oracle, response fluency becomes a misleading substitute for actual completion.
Before modeling
Write down the person's request, the current state, and the evidence that will prove completion.
Starting error
Starting with a function name or a button produces a tool without a clear human goal.
Anticipate before opening the code
What information changes the decision? What effect needs to appear in the interface? Answer before implementing.
expected: { enrollmentStatus: "draft" }
forbidden: { chargeCreated: true }Going deeper: evidence worth keeping
Record the input, previous state, decision made, result, and visible change to the page.
Also include a simulated failure and the observed behavior without WebMCP support.
Key Concepts
initial state
expected effect
forbidden effect
oracle
Build a representative dataset
What is it: Cases cover frequency, risk, ambiguity, failures, and real-world language variation.
Why learn this: An average dominated by easy cases hides critical regressions.
| Question | Strong contract | Weak contract |
|---|---|---|
| When should you use it? | The description names the intent and context. | “Does things” or “manages.” |
| What does it receive? | Only the data needed. | Generic, unlimited object. |
| What does it return? | Verifiable state and next action. | Text without a protocol. |
| How does it fail? | Code, reason, and recovery. | Opaque exception. |
Do
- ✓ Use specific verbs.
- ✓ Define the inputs.
- ✓ State the evidence.
Avoid
- ✗ Mix intents.
- ✗ Accept any object.
- ✗ Hide effects.
dataset = stratify(cases, ["happy", "ambiguous", "failure", "adversarial"]);Going deeper: evidence worth keeping
Record the input, previous state, decision made, result, and visible change to the page.
Also include a simulated failure and the observed behavior without WebMCP support.
Key Concepts
strata
risk weighting
paraphrase
edge case
Evaluate each pipeline stage
What is it: Discovery, selection, arguments, policy, execution, and response each have their own metrics.
Why learn this: The breakdown pinpoints the regression and guides the fix.
Observe
Capture state, input, and context before taking action.
Decide
Validate preconditions and choose the allowed transition.
Run
Apply the shared rule and honor cancellations.
Prove
Update the interface and return a structured result.
Checkpoint
If step 2 fails, the tool doesn’t try to “make it work”: it returns an explicit recovery path.
metrics = { discoveryRecall, correctTool, validArgs, policyCompliance, taskSuccess };Going deeper: evidence worth keeping
Record the input, previous state, decision made, result, and visible change to the page.
Also include a simulated failure and the observed behavior without WebMCP support.
Key Concepts
retrieval recall
tool accuracy
schema validity
task success
Instrument traces without leaking data
What is it: Spans correlate turns and calls with sanitized attributes, versions, and duration.
Why learn this: Useful observability preserves diagnostic value without storing unnecessary prompts or PII.
REFERENCE CODE
span.setAttributes({ toolName, catalogVersion, outcome, durationMs });
span.addEvent("policy.denied", { reasonCode });It is small, documented, and verifiable.
Reuses the application's actual rule.
Lets you verify the effect and continue.
Read the code as a contract
Underline where input is validated, where the effect happens, where cancellation arrives, and where the UI is updated.
Going deeper: evidence worth keeping
Record the input, previous state, decision made, result, and visible change to the page.
Also include a simulated failure and the observed behavior without WebMCP support.
Key Concepts
OpenTelemetry
span
redaction
sampling
Detect regressions and drift
What is it: Versioned baselines let you compare models, prompts, catalogs, and browsers over time.
Why learn this: Changes that seem isolated can alter selection and success rates.
Simulated failure
Run the scenario with missing state, invalid input, or unavailable capacity.
Expected recovery
The response explains what failed, what remains safe, and what action can continue.
Minimum test matrix
✓ reproducible happy path
✓ actionable invalid input
✓ cancellation ends the work
✓ fallback preserves the user journey
✓ interface reflects the state
✓ backend retains authorization
gate.failIf(taskSuccess < baseline - 0.02 || unauthorizedEffects > 0);Going deeper: evidence worth keeping
Record the input, previous state, decision made, result, and visible change to the page.
Also include a simulated failure and the observed behavior without WebMCP support.
Key Concepts
baseline
confidence interval
drift
canary
Turn evaluation into a release gate
What is it: Minimum risk-based criteria block publication, while noncritical alerts trigger follow-up.
Why learn this: Evals go from decorative reports to operational controls.
Synthesis exercise
- 1. Explain the problem without using the name of the technology.
- 2. Design the before and after states.
- 3. Implement the smallest working proof.
- 4. Trigger a failure and record the recovery.
- 5. Deliver code, evidence, and one known limitation.
Module challenge
Build an eval harness with happy-path, ambiguous, and adversarial cases, and publish a release decision dashboard.
Responsible Expert criterion
The implementation must work, explain its limitations, and preserve the human path when the experimental capability is unavailable.
release.requires([security.zeroUnauthorized, p95Latency.lt(3000), criticalTasks.gte(0.98)]);Going deeper: evidence worth keeping
Record the input, previous state, decision made, result, and visible change to the page.
Also include a simulated failure and the observed behavior without WebMCP support.
Key Concepts
quality gate
risk tier
waiver
evidence
📦 Module delivery
Build an eval harness with happy-path, ambiguous, and adversarial cases, and publish a release decision dashboard.
Acceptance criteria
The implementation works with WebMCP support and preserves the manual path when the API is unavailable.
Evidence
Include code, the tested scenario, the observed result, and one known limitation.
Technical sources
- Draft and official WebMCP repository
- Rendered specification
- Official snapshot: 26/08/2026 · commit 41d12f0. The declarative API remains experimental; validate the draft before production.