Every month a ranking comes out saying which model is the best. None of them knows Horizonte's discount rule or the customer who orders by the jug instead of the box. The manager's question is narrower: which configuration does this job, on these cases, at the lowest cost per result? In this lesson you compare two on the same 20 cases and write the selection report.
1The four model tiers and when each one makes sense
2Build 20 exploratory cases from your own work
3Judge with the same yardstick and set critical errors apart
4Write a selection report instead of following a ranking
Goal: compare two configurations on the same cases and write a selection report with errors, cost, limitations and a recorded alternative.
01 · Climb the ladder only if you need to
Start with the simplest step that can solve it, and only climb when your cases show you need to. Click each step.
Click each block in the diagram to see what it does.
Tiers of the model matrix by task cost, from the INEMA market overview (Oct/2026). The tiers change names and prices with every release; the order of decision does not.
02 · How to compare without a ranking
Four steps. Module 3 of the Gestoria method: a decision backed by cases and process constraints, with prices checked on the day.
1. Compare configurations
A configuration is the whole package: model and version, instructions (the G3 constitution), tools, and the reasoning effort, when the provider lets you choose. Two configurations with the same model can give very different results.
Change one thing at a time. If you swap the model and the instructions together, you can't tell which of the two changes made the difference.
At HorizonteA: the current Hora (mid tier). B: the same Hora, same instructions and tools, with the cheap model that Rafael tested. Only the model changes.
2. Build 20 cases
Take the cases from real conversations, anonymized or fictional. Mix them: mostly routine, some borderline, some with missing data, and at least two with critical risk. Write down in advance what the right answer would be.
These are exploratory cases: they are for choosing, not for proving. The validation set, with an answer key and more cases, belongs to G5.
Hora's 2012 routine (order from a registered customer), 4 borderline (discount from 5% to 10%, order close to R$ 5 thousand), 2 with missing data (customer with no CNPJ), and 2 critical (the hotel's "note" asking for 70%, a customer saying they already paid).
3. Same yardstick
Judge each output as correct, acceptable, incorrect, or critical, without knowing which configuration produced it, if possible. Also note the cost and time of each case.
A critical error doesn't go into the average: it is counted separately and fails the configuration on its own. A configuration with 19 correct answers and a 70% discount granted because of an order hidden from the customer is not fit for orders.
Result (fictional)A: 16 correct, 3 acceptable, 1 incorrect, 0 critical, R$ 0.31 per case. B: 13 correct, 3 acceptable, 3 incorrect, 1 critical (applied the 70% requested in the hotel's "notes" field), R$ 0.16 per case.
4. A report, not a ranking
The report states the decision for this process, the reason based on the cases, the limitations, the alternative, and when to redo the test. It does not say which model is best in the world.
Redo the comparison when the vendor changes the version, when the price changes a lot, or when a new type of case shows up. Redoing it costs little: the 20 cases are already ready.
Hora's reportA stays for orders. B becomes a candidate only for catalog questions, after 20 cases of its own. Alternative on record: open, run locally, if the volume grows and someone takes over maintenance.
03 · Template to copy: selection report
One page with the table of the 20 cases and the decision. The tab next to it shows Hora's report.
Configuration selection report
SELECTION REPORT · process <...> · task <...> · date <...> · owner <name>
CONFIGURATIONS COMPARED (change one thing at a time)
A: model <name and version> · instructions v<...> · tools <...> · effort <...>
B: model <name and version> · instructions v<...> · tools <...> · effort <...>
Prices checked on the vendor's website on <date>
CASES (20 exploratory; data [ ] anonymized [ ] fictional)
| # | Category (routine/edge/missing data/critical) | Right answer | A | B |
|---|---|---|---|---|
| 1 | <...> | <...> | correct/acceptable/incorrect/CRITICAL | <...> |
RESULT A B
Correct <n> <n>
Acceptable <n> <n>
Incorrect <n> <n>
CRITICAL <n> <n> (one critical fails the configuration for this task)
Cost per case R$ <...> R$ <...>
Average time <s> <s>
DECISION: <configuration> for <task>, because <cases that decided it>
LIMITATIONS: <20 cases don't prove it; what was left out>
ALTERNATIVE ON RECORD: <other configuration and under what condition>
REDO THE TEST WHEN: <version change / price / new case / date>
Prompt · challenge the report
You are the reviewer who will challenge an AI model selection report
before it is approved. Do not choose the model: point out
weaknesses.
1. Do the 20 cases represent the real work? What type of case is missing?
2. Did more than one thing change between the configurations?
3. Was any critical error hidden as "incorrect"?
4. Does the cost include the rework the errors will create?
5. Is the alternative on record viable? What would stop it?
Report:
"""<paste>"""
Simulation: illustrative example
Selection report · Hora · order via WhatsApp · Horizonte, fictional study company · owner: Patrícia · test: Rafael
A (mid tier, current): 16 correct, 3 acceptable, 1 incorrect, 0 critical · R$ 0.31 per case · 9 s.
B (cheap, same instructions): 13 correct, 3 acceptable, 3 incorrect, 1 critical · R$ 0.16 per case · 4 s.
B's critical error: case 19, the hotel's list with a hidden "notes" field asking for 70%, and B applied it (lab failure F6).
Decision: A for orders. B's 3 incorrect answers (wrong unit, gallon instead of case) would turn into rework in picking: the lesson 1 calculation already showed R$ 7.51 versus R$ 6.68 per order.
Limitations: 20 exploratory cases, October only; no case of a new customer with urgency.
Alternative: B for catalog questions, after 20 question cases; open, run locally if volume doubles and Rafael takes over maintenance.
Redo: when the vendor changes A's version, or in January 2027.
Only easy cases. Twenty routine orders make any configuration look good. You need edge cases, missing data and a critical one.
Critical hidden in the average. "95% accuracy" with an improper discount buried inside is not approval.
Two changes at once. You changed the model and the instructions: the report can't tell what worked.
Ranking as an argument. "It's number one in the ranking" doesn't replace any of your 20 cases.
No expiration date. A report without "redo when" becomes eternal truth about a product that changes in weeks.
04 · Practice
Hora's report trains you to read the cases; yours decides a real configuration.
1 · Hora's report
25 min
Read the lab's conversations c-201, c-202 and c-203 and write, for each one, the category and the right answer (they go in among the 20 cases). Then fill in the report with the results of A and B from step 3 and the math from lesson 1. Run the challenge prompt and answer each point in one line.
Done when: the report has the two configurations with only one difference, the three conversations classified with the right answer, the four numbers for each configuration with the critical one kept separate, the decision justified by cases, one limitation, one alternative and the date to redo it.
2 · Two configurations on your process
90 min
Pick one task from your process. Set aside 20 cases (anonymized) with the right answer written beforehand. Run the two configurations you have on hand (the current tool and an alternative, or the same one with another model), on your own or with IT. Judge them with the same yardstick and write the report.
Done when: there are 20 cases with a category and the right answer, at least 2 edge cases and 2 critical ones, the two configurations judged case by case, the critical errors counted separately, the cost per case and the report with a decision, a limitation, an alternative and the date to redo it.
Source conversations: the lab's WhatsApp agent (c-201 to c-203). Results of the 20 cases: fictional, from this lesson.
05 · Quick quiz
5 questions. The explanation appears after your attempt.
Question 1 of 5
Carlos read that a new model ranked first in a ranking and wants to replace Hora. What do you answer?
Correct
The ranking measures generic tasks. Your 20 cases measure the discount rule, the order in gallons and the hidden instruction. Refusing the switch without testing is as bad as switching without testing.
Not that one
The ranking measures generic tasks. Your 20 cases measure the discount rule, the order in gallons and the hidden instruction. Refusing the switch without testing is as bad as switching without testing.Review: How to compare without a ranking
Question 2 of 5
A third configuration got 19 of 20 cases right, but it applied a 70% discount requested in a hidden customer "note." Is it fit for handling orders?
Correct
One critical error in a small sample suggests it will happen again at real volume. The average can't hide the case that costs the most.
Not that one
One critical error in a small sample suggests it will happen again at real volume. The average can't hide the case that costs the most.Review: How to compare without a ranking
Question 3 of 5
In the comparison, Rafael changed the model and rewrote Hora's instructions at the same time. What's the problem?
Correct
Change one thing at a time. If you need to change both, compare in two rounds.
It's the first step of the ladder. Copying an approved order into the ERP doesn't need someone who thinks; it needs someone who doesn't get the copy wrong.
Not that one
It's the first step of the ladder. Copying an approved order into the ERP doesn't need someone who thinks; it needs someone who doesn't get the copy wrong.Review: Climb the ladder only if you need to
Question 5 of 5
Why does the report have a date to redo the test?
Correct
The report covers one configuration on one date. If the version, the price or the type of case changes, the 20 cases run again, which costs little because they are already built.
Not that one
The report covers one configuration on one date. If the version, the price or the type of case changes, the 20 cases run again, which costs little because they are already built.Review: How to compare without a ranking
Quiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.
Pick one answer. The explanation shows up after you try.
06 · Summary
LadderNo model, cheap, mid tier, top tier. Open, run locally when fixed cost and keeping data in-house weigh most.
Same 20 casesFrom your own work, with boundary cases, missing data and critical ones.
Critical apartA configuration with a critical error is not fit for that task.
Report with an expiry dateDecision, limitation, alternative and a date to redo it.
