Verify with evidence
Build objective tests and rubrics that explain the editorial evaluation.
What it is
An objective check has a reproducible rule: the file exists, the sum reconciles, and the required field is present. An editorial assessment compares the output against a rubric: clarity, relevance, and suitability for the audience. The same requirement can combine both.
Why learn
The video uses review as the central step. To apply it well, avoid confusing “another agent looked at it” with factual proof. Image counts are verifiable; for an image to truly explain an idea requires interpretation and localized evidence.
Key concepts
- Goal: a rule and an observable result.
- Editorial: criteria and anchored examples.
- Hybrid: measurement plus inspection.
- Evidence: file, excerpt, screenshot, or test.
What it is
A useful test should fail in the face of a plausible error. If it only checks that the file exists, an empty report will pass. In the example, compare the total, the channels, the number of records, and the behavior for invalid input.
Why learn
Tests derived blindly from the implementation can repeat the same error. Use at least one result computed by another path: manually in a small case, or by an independent rule. Also verify outputs that should not be produced.
Key concepts
- Assertion: the expected condition.
- Oracle: the source of the correct result.
- Negative test: handled error.
- Independence: reduce shared error.
What it is
A rubric explains what each quality level means. Use a few criteria and concrete descriptions. For clarity, for example: level 0 is contradictory, 1 requires guessing, 2 is understandable with review, and 3 is straightforward and self-sufficient.
Why learn
Without anchors, two reviewers can use the same word for different standards. An isolated high score doesn’t help you fix anything. Require the evaluation to cite the excerpt and propose the smallest change that can resolve the problem.
Key concepts
- Criterion: the assessed dimension.
- Anchor: an example of the level.
- Excerpt: the location of the evidence.
- Correction: a limited and verifiable action.
What it is
An LLM as a judge applies criteria and points to possible failures. It can interpret language and images, but it can also miss errors or approve its own hypothesis. A second instance doesn’t guarantee independence of reasoning.
Why learn
Separate generation from the evaluation and provide the evaluator with the contract, the input, and the output. When possible, hide which configuration produced each version. Check a human sample to calibrate whether the judge is actually using the rubric.
Key concepts
- Separation: review has its own context and purpose.
- Blinding: reduces preference for labels.
- Calibration: compare with human judgment.
- Limit: the judge doesn’t replace the primary source.
What it is
A quality report connects criterion, result, and evidence. “Everything verified” doesn’t say what was opened, calculated, or inspected. A good delivery also includes limitations and items that still need a decision.
Why learn
This document reduces human review work because it directs attention. But QA itself can also be wrong: open samples of the evidence and check whether what was described actually happened.
Key concepts
- Status: passed, failed, or not verified.
- Evidence: a locally referencable reference.
- Action: correction or follow-up.
- Traceability: input and version used.
What it is
The cycle is produce, verify, correct, and verify again. Set a cap on attempts and an exit condition. In the lab, use up to two rounds of correction before reporting a persistent block for analysis.
Why learn
More rounds don’t imply better quality. The agent may alternate between two solutions or spend resources without improving. A cap makes the cost predictable and forces you to explain what’s missing instead of declaring victory from exhaustion.
Key concepts
- Acceptance: all blockers passed.
- Correction: focus on the observed failure.
- Regression: what worked continues working.
- Stop condition: limit or external dependency.