The three primitives
Write questions that can be evaluated.
- Choice: one alternative per question
- Score: an ordinal scale
- Noul: the strength of an assertion
- Confidence is not probability of being correct
- The JSON contract before the model
- Exercise: formulate and label
Choice: one alternative per question
What it is
Choice returns the alternative with the highest value in the distribution, and the probabilities of the options. In this project, billing covers payments, technical covers product issues, sales covers new purchases, and other receives cases that don’t fit. The descriptions accompany the names in the encoder’s internal prompt.
Why learn
Short names with clear descriptions help distinguish classes without using up the whole token budget. If you need to assign multiple independent labels, a single forced choice doesn’t express the problem; consider separate binary questions and evaluate each one.
An error-login message corresponds to technical, while a request for a duplicate invoice corresponds to billing.
✓ Apply with criteria
Use two independent noul questions. Choice is suitable for selecting a main destination, not for representing all signals of a message.
✗ Avoid the automatic conclusion
How do you classify both refund and the threat to cancel at the same time?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
highest value
set of probabilities
class description
other option
Test your understanding
How do you classify both refund and the threat to cancel at the same time?
Check the commented answer
Use two independent noul questions. Choice is suitable for selecting a main destination, not for representing all signals of a message.
Score: an ordinal scale
What it is
Score uses an ordered list of levels. The output is the distribution’s expectation over the indices: with levels 0, 1, and 2, you might see 1.84. This does not represent a new class or a deadline in hours. The legend must explain what each rung means for operations.
Why learn
A scale only makes sense when the levels have order and distinguishable criteria. Normal, time-sensitive, and immediate blocking are a starting rubric. The upstream points to score as a hard primitive; don’t turn a small decimal difference into an irreversible decision.
With probabilities 0.10; 0.30; 0.60, the score is 0×0.10 + 1×0.30 + 2×0.60 = 1.50.
-
1
Observe
With probabilities 0.10; 0.30; 0.60, the score is 0×0.10 + 1×0.30 + 2×0.60 = 1.50.
-
2
Define
Score uses an ordered list of levels.
-
3
Check
No. It’s an average that can come from different distributions. Check the probabilities per level and the text before defining priority.
Key concepts
ordered levels
weighted average
meaning of the levels
mean absolute error
Test your understanding
Does a score of 1.5 prove the occurrence is critical?
Check the commented answer
No. It’s an average that can come from different distributions. Check the probabilities per level and the text before defining priority.
Noul: the strength of an assertion
What it is
Noul answers an assertion with a value between zero and one. The question must be observable in the state: does the customer ask for their money back? Does the customer threaten to cancel? The implementation uses false and true options and returns the probability of true. It’s not authorization to act.
Why learn
Questions about explicit intent are easier to label than vague predictions about future behavior. In this lab, churn_risk means a textual threat to cancel; it wasn’t trained or validated here to predict real cancellations over months.
I want my money back contains an explicit request. I’m dissatisfied does not necessarily mean the person will cancel.
Example of reasoning
"department": {
"type": "choice",
"instructions": "Which department handles this message?",
"criteria": {"billing": "payments and refunds",
"technical": "bugs and outages",
"sales": "new purchases",
"other": "unrelated or insufficient information"}
}
Key concepts
proposition
modeled probability
observed phrase
operational cut
Test your understanding
Is the name churn_risk enough to make the value a business forecast?
Check the commented answer
No. The definition of the question restricts the meaning: an explicit threat signal. Predicting real churn would require a different dataset and longitudinal validation.
Confidence is not probability of being correct
What it is
In the SDK analyzed, the confidence of choice and score is 1 minus the normalized entropy of the distribution. Concentrated distributions produce high values; spread-out distributions produce low values. Noul uses the largest value between P(true) and P(false). These fields don’t have exactly the same definition.
Why learn
Don’t calculate ECE using a field called confidence without checking the code. For classification evaluation, the project uses the probability of the chosen class. Even that value must be compared with the actual hit frequency in separate data.
A very concentrated distribution on the wrong option will have high confidence. An inadequate checkpoint for the language can cause this kind of error.
✓ Apply with criteria
No. By the normalized entropy formula, confidence is zero. The highest probability is still 0.5; these are different measurements.
✗ Avoid the automatic conclusion
Two classes each with 0.5 have choice confidence of 0.5?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
dispersion
highest probability
frequency versus prediction
internal certainty
Test your understanding
Two classes each with 0.5 have choice confidence of 0.5?
Check the commented answer
No. By the normalized entropy formula, confidence is zero. The highest probability is still 0.5; these are different measurements.
The JSON contract before the model
What it is
The schema contains stable identifiers, type, instructions, and criteria when needed. Choice uses a dictionary; score uses an ordered list; noul can use an assertion without extra criteria. Keeping stable identifiers preserves API consumers when wording improves.
Why learn
Validate results before the policy: classes must be known, values must be finite, and probabilities should approximately sum to one. SDK rounding requires a small tolerance. Invalid inputs must produce a visible error, not a response that looks like it was calculated.
A response with NaN, an unknown department, or an incomplete distribution is rejected by the adaptation.
Key concepts
stable format
data checking
rounding
code that reads output
Test your understanding
Why not silently accept a new category?
Check the commented answer
It may not have a defined destination. The schema, policy, interface, and tests must be updated together before using the new category.
Exercise: formulate and label
What it is
Write six support messages and answer the four questions manually. Include duplicate billing, technical failure, new purchase, irrelevant conversation, explicit cancellation, and dissatisfaction without cancellation. Ask the other person to apply the same rubric without seeing your answers.
Why learn
Disagreements between humans show where the question or the rubric needs improvement. Don’t use the model to automatically decide which human is correct. Record the rule you adopt and keep some examples that won’t be used during the question refinement.
Ambiguous case: the system crashed and I want a refund. The main destination depends on the rule; the refund_requested signal can be positive regardless of it.
Key concepts
human answer
multiple readings
documented agreement
unaligned examples
Test your understanding
What should accompany each labeled example?
Check the commented answer
An identifier, the message, the expected label, and, in a separate dataset, the justification and the rubric version. Remove unnecessary personal data.
Module summary
- Use two independent noul questions. Choice is suitable for selecting a main destination, not for representing all signals of a message.
- No. The definition of the question restricts the meaning: an explicit threat signal. Predicting real churn would require a different dataset and longitudinal validation.
- It may not have a defined destination. The schema, policy, interface, and tests must be updated together before using the new category.
Select a snippet from the lesson to highlight or annotate. Questions and notes stay in your journey; export the JSON to back up.