Separate objective checks from judgment
What it is
An objective check has a reproducible rule: the file exists, the sum reconciles, and the required field is present. An editorial assessment compares the output against a rubric: clarity, relevance, and suitability for the audience. The same requirement can combine both.
Why learn
The video uses review as the central step. To apply it well, avoid confusing “another agent looked at it” with factual proof. Image counts are verifiable; for an image to truly explain an idea requires interpretation and localized evidence.
Key concepts
Objective: 4 records and total 500.00.
Editorial: the summary lets you understand the main finding.
Hybrid: the table includes all channels and is readable on a cell phone.
✓ Do it like this
Match each item to a concrete piece of evidence.
✗ Avoid this mistake
Classify all visual review as subjective; overflow is measurable.
Practice before revealing
“The image has 1200 pixels of width” proves that it’s appropriate?
View commented response
No. It’s an objective property, but it doesn’t show relevance or readability of the content. You still need to assess the crop and the context.
Build tests that can fail
What it is
A useful test should fail in the face of a plausible error. If it only checks that the file exists, an empty report will pass. In the example, compare the total, the channels, the number of records, and the behavior for invalid input.
Why learn
Tests derived blindly from the implementation can repeat the same error. Use at least one result computed by another path: manually in a small case, or by an independent rule. Also verify outputs that should not be produced.
Key concepts
assert total == "500.00"
assert canais == {"Loja": "320.00", "Site": "180.00"}
assert registros == 4
# Input without a column: output code != 0
# And no relatorio.md new item should appear.
From concept to action
- Known input: identify the initial condition.
- Execution: apply the described decision.
- Compare with oracle: check the effect in the example.
- Pass or fail: record the output evidence.
✓ Do it like this
Introduce an error on purpose to verify that the test flags it.
✗ Avoid this mistake
Use “the script ran” as the only quality criterion.
Practice before revealing
If the per-channel totals add up to 490 and the overall total shows 500, what went wrong?
View commented response
Reconciliation. Even if each field is present, the numbers are not consistent. The output must be corrected before it’s accepted.
Write an editorial rubric with anchors
What it is
A rubric explains what each quality level means. Use a few criteria and concrete descriptions. For clarity, for example: level 0 is contradictory, 1 requires guessing, 2 is understandable with review, and 3 is straightforward and self-sufficient.
Why learn
Without anchors, two reviewers can use the same word for different standards. An isolated high score doesn’t help you fix anything. Require the evaluation to cite the excerpt and propose the smallest change that can resolve the problem.
Key concepts
Image relevance
0: contradicts the text.
1: is from the video, but does not illustrate the statement.
2: illustrates, but needs additional caption.
3: illustrates with context and clearly legible detail.
✓ Do it like this
Ask for justification per criterion, with an example from the model’s own output.
✗ Avoid this mistake
Ask only for “give a score from 0 to 10”.
Practice before revealing
Provide a failure anchor for the criterion “fidelity to the data”.
View commented response
“Any number or cause presented without support in the input is a blocking failure.” Some dimensions should not be compensated by averaging with style.
Use an evaluator without outsourcing the truth
What it is
An LLM as a judge applies criteria and points to possible failures. It can interpret language and images, but it can also miss errors or approve its own hypothesis. A second instance doesn’t guarantee independence of reasoning.
Why learn
Separate generation from the evaluation and provide the evaluator with the contract, the input, and the output. When possible, hide which configuration produced each version. Check a human sample to calibrate whether the judge is actually using the rubric.
Key concepts
Evaluate this output against the provided rubric.
For each failure, cite the excerpt, the criterion, and the evidence.
Do not assume the previous QA report is correct.
If you cannot verify an item, mark it “not verified.”
✓ Do it like this
Ask the judge to look for contradictions and missing information.
✗ Avoid this mistake
Request that the judge “confirm that it’s perfect”.
Practice before revealing
How do you reduce the influence of the model name in the comparison?
View commented response
Present versions A and B with alternating order and the same rubric. Keep the mapping to the models separate until the evaluation ends.
Produce a useful QA report
What it is
A quality report connects criterion, result, and evidence. “Everything verified” doesn’t say what was opened, calculated, or inspected. A good delivery also includes limitations and items that still need a decision.
Why learn
This document reduces human review work because it directs attention. But QA itself can also be wrong: open samples of the evidence and check whether what was described actually happened.
Key concepts
Criterion: total sum
Status: passed
Expected: 500.00 | obtained: 500.00
Evidence: dados/vendas.csv, four lines
Limitation: there was no reconciliation with an external system.
From concept to action
- Criterion: identify the initial condition.
- Result: apply the described decision.
- Evidence: check the effect in the example.
- Limitation: record the output evidence.
✓ Do it like this
Record the verification command or method.
✗ Avoid this mistake
Say you opened the browser without having used a navigation tool.
Practice before revealing
Write the status of a visual review when only HTML was inspected.
View commented response
“Not verified visually; HTML structure inspected, rendering pending.” The report must distinguish what was observed from what was inferred.
Close a review cycle with a limit
What it is
The cycle is produce, verify, correct, and verify again. Set a cap on attempts and an exit condition. In the lab, use up to two rounds of correction before reporting a persistent block for analysis.
Why learn
More rounds don’t imply better quality. The agent may alternate between two solutions or spend resources without improving. A cap makes the cost predictable and forces you to explain what’s missing instead of declaring victory from exhaustion.
Key concepts
Round 1: correct total, missing period → fix the header.
Round 2: check the period and repeat the sum test.
Did the failure persist? Provide diagnosis and evidence.
Do not mark it as approved.
✓ Do it like this
Run the check again after the correction.
✗ Avoid this mistake
Edit the output and reuse a prior QA before the change.
Practice before revealing
Did the report change after verification? Does the QA still hold?
View commented response
Only for parts proven to be unaffected. The final delivery must be verified in the final state, especially against criteria linked to the change.
Check your understanding
A second agent says “it’s correct”. Does that prove the facts?
What you take from this module
Write a traceable rubric and QA report.
- Separate objective checks from judgment.
- Build tests that can fail.
- Write an editorial rubric with anchors.
- Use an evaluator without outsourcing the truth.
- Produce a useful QA report.
- Close a review cycle with a limit.
Next action: save the exercise in your learning lab and record what still needs review.