Benchmarks, calibration, and evaluation
Compare what was actually measured.
- Read the comparison unit
- The limits of zero-shot
- Accuracy and confusion matrix
- Brier and ECE answer different questions
- Calibration requires reserved data
- Run and interpret the local evaluation
Read the comparison unit
What it is
The sources combine results from different benchmarks, tasks, and checkpoints. The performance of a specialized model does not describe the base model. The Jev numbers cited by Laya come from third parties and not from a controlled run by the same author using identical prompts.
Why learn
Before repeating a percentage, note the model, the revision, the dataset, the number of classes, the number of examples, and the method. A table can be useful as an indication for investigation without supporting a universal ranking. The conditions must accompany the numbers in the report.
Typed-decisions uses specific workflows; Banking77 tests many classes. Winning the first does not imply winning the second.
✓ Apply with criteria
Not without caveats. The answer space and the sample change the difficulty; record the difference and avoid attributing everything to the architecture.
✗ Avoid the automatic conclusion
Can I compare 72 classes with 77 as if it were the same test?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
conditions
used examples
measured objective
aligned conditions
Test your understanding
Can I compare 72 classes with 77 as if it were the same test?
Check the commented answer
Not without caveats. The answer space and the sample change the difficulty; record the difference and avoid attributing everything to the architecture.
The limits of zero-shot
What it is
The current report shows base checkpoints below the major-class baseline in typed-decisions, while the specialized version improves. This suggests that instructing new questions does not guarantee competence in the domain. Specialization and training data matter for interpreting the result.
Why learn
In the lab, the small set of tickets is a functional smoke test. Even an excellent result does not validate real messages, abbreviations, adversarial cases, or rare categories. The next step is to gather representative examples before deciding to invest in training.
A model trained for four synthetic workflows is not automatically a universal analyst of business processes.
-
1
Observe
A model trained for four synthetic workflows is not automatically a universal analyst of business processes.
-
2
Define
The current report shows baseline checkpoints below the majority class baseline in typed-decisions, while the specialized version improves.
-
3
Check
No. The sheet specifies the specific scope and recommends your own validation; performance outside those workflows may drop.
Key concepts
task without fine-tuning
simple reference
fine-tuned model
new data
Test your understanding
If the specialized model has good accuracy, does that remove the need to test in another domain?
Check the commented answer
No. The sheet specifies the specific scope and recommends your own validation; performance outside those workflows may drop.
Accuracy and confusion matrix
What it is
Accuracy is the fraction of correct predictions. It can hide imbalance and errors concentrated in a single class. A confusion matrix, for each expected label, shows which outputs were produced. Also compare a simple rule, such as always choosing the most frequent class.
Why learn
In support, confusing sales with other can have a different impact than diverting a critical incident. Report errors by category and representative examples. With a small sample, show counts in addition to percentages so you don’t suggest statistical precision that the data doesn’t support.
If 90 out of 100 tickets are financial, choosing billing will always give 90% accuracy, without understanding any text at all.
Example of reasoning
python3 -m practical --device cuda evaluate --output avaliacao.json
# Local sample: 16 synthetic tickets, 4 departments
# 13/16 correct; majority baseline = 4/16
# Educational result, not production validation.
Key concepts
correct fraction
error between classes
unequal classes
dominant class
Test your understanding
Why does the training dataset include four classes in equal quantities?
Check the commented answer
To make the smoke test easy to inspect and to make the baseline explicit. This does not necessarily reproduce the real frequency of an operation.
Brier and ECE answer different questions
What it is
The multiclass Brier score sums the squared error between probabilities and the one-hot label. ECE compares, in probability bins, average confidence and accuracy frequency. The implementation uses the probability of the chosen class in ten bins, not the SDK’s entropy-based confidence.
Why learn
State the formula: some reports divide Brier by the number of classes, others sum it. ECE depends on the bins and the sample; with few cases, bins may be empty or unstable. A small value in sixteen examples is not proof of calibration.
An incorrect prediction with probability 0.99 gets a larger penalty than a cautious distribution. Both can have the same argmax accuracy.
✓ Apply with criteria
Normalizations, classes, and datasets may differ. Even the same metric name does not guarantee an identical definition.
✗ Avoid the automatic conclusion
Why not compare Brier from two reports without reading the method?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
error of the distribution
difference by bins
single label
probability interval
Test your understanding
Why not compare Brier from two reports without reading the method?
Check the commented answer
Normalizations, classes, and datasets may differ. Even the same metric name does not guarantee an identical definition.
Calibration requires reserved data
What it is
Temperature scaling divides logits by a temperature before applying softmax. It can reduce overconfidence without improving the chosen classification. Adjusting the temperature and measuring its effect on the same data creates an optimistic estimate. Separate calibration from the final evaluation.
Why learn
The multilingual checkpoint is described as not calibrated for the user’s domain. The lab reports this condition and does not pretend to perform calibration. Before trusting a threshold, gather enough data and track error rate versus coverage on the reserved set.
Training teaches parameters; calibration adjusts probabilities; testing measures the final result without new adjustments.
Key concepts
logits scale
reserved data
accepted fraction
test misuse
Test your understanding
Can adjusting temperature correct the wrong department in all cases?
Check the commented answer
No. A single positive global temperature preserves the order of logits. It changes the distribution, not the ability to distinguish classes.
Run and interpret the local evaluation
What it is
The evaluate command reads JSONL, runs real weights, and writes per-example results—accuracy, baseline, Brier, and ECE. The messages are synthetic, and the report includes an explicit limitation. Inspect each error before thinking about training or changing a threshold.
Why learn
Use the report as a starting point for your own set. Add negations, duplicates, very short messages, rare categories, and long texts. If you change questions after seeing the results, keep the previous version and evaluate on new reserved cases.
python3 -m practical --device cuda evaluate --output avaliacao.json
Key concepts
one case per line
results per case
a diagnostic opportunity
unseen data
Test your understanding
What conclusion is allowed if all the didactic examples are correct?
Check the commented answer
That the inference path classified that sample correctly, in that environment and scheme. Not that the tool is validated for production.
Module summary
- Not without caveats. The answer space and the sample change the difficulty; record the difference and avoid attributing everything to the architecture.
- To make the smoke test easy to inspect and to make the baseline explicit. This does not necessarily reproduce the real frequency of an operation.
- No. A single positive global temperature preserves the order of logits. It changes the distribution, not the ability to distinguish classes.
Select a snippet from the lesson to highlight or annotate. Questions and notes stay in your journey; export the JSON to back up.