Specialization and final project
Define when to evolve and when to stop.
- Start with the error that matters
- Build data with clear separation
- The training notebook is a reference
- RLCD and scoring rules
- Gradual and reversible operation
- Final delivery: a process-explainable decision
Start with the error that matters
What it is
Fine-tuning has a cost and only makes sense when the errors you find justify changing the model. Some problems come from ambiguous categories, incomplete state, or an inadequate checkpoint. Fixing those causes can be cheaper and more effective than training a new network.
Why learn
Classify errors before acting: human-label errors, schema errors, context errors, language errors, or capacity errors. Preserve examples and execution conditions. Then choose a small intervention and evaluate again, without changing multiple variables at the same time.
If the state does not include the final request, expanding training does not recover missing information.
✓ Apply with criteria
An ambiguous operational contract or missing information. First define the desired outcome and provide the necessary facts.
✗ Avoid the automatic conclusion
Which triage problem is not solved just by fine-tuning?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
likely cause
controlled change
needed learning
total effort
Test your understanding
Which triage problem is not solved just by fine-tuning?
Check the commented answer
An ambiguous operational contract or missing information. First define the desired outcome and provide the necessary facts.
Build data with clear separation
What it is
A domain dataset needs to represent language, class frequencies, and difficult cases of the operation. Split training, calibration, and testing by groups that avoid leakage, such as conversation or customer. Almost-duplicate messages in different parts can inflate results.
Why learn
Record the rubric version and labeling process. When the label comes from a teacher model, treat it as imperfect supervision. Agreement with the teacher measures alignment with that reference, not universal truth or guaranteed quality for real customers.
A conversation with five messages should remain in the same split, so the test does not reveal only excerpts that were already seen.
-
1
Observe
A conversation with five messages should remain in the same split, so the test does not reveal only excerpts that were already seen.
-
2
Define
A domain dataset needs to represent language, class frequency, and the operation’s difficult cases.
-
3
Check
Messages from the same conversation or duplicates may appear in both training and test, making the test artificially easy.
Key concepts
partition
separation unit
label source
shared information
Test your understanding
Why can randomly splitting each message be inadequate?
Check the commented answer
Messages from the same conversation or duplicates may appear in both training and test, making the test artificially easy.
The training notebook is a reference
What it is
The repository includes a fine-tuning notebook for typed-decisions on two T4 GPUs. It prepares data, trains, adjusts temperatures, and evaluates. The published time is the author’s observation for that environment and setup, not a guaranteed estimate for your machine or dataset.
Why learn
Read the cells, the output checkpoints, and the publish destination before running. The course does not run this training automatically or announce a trained model. The checkpoint used in the lab is the multilingual one provided by the upstream.
The notebook is located at notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb in the practical repository.
Example of reasoning
# Experiment plan — does not run training.
1. Version the rubric and examples
2. Split training / calibration / test per conversation
3. Measure baseline and current model
4. Adjust only on the allowed set
5. Evaluate the reserved test
6. Approve or reject the version using predefined criteria
Key concepts
cell-by-cell execution
data pass
new weights
separate stage
Test your understanding
Does this course deliver a fine-tuning already done for your business?
Check the commented answer
No. It delivers the adapted runtime, didactic evaluation, and an evolution roadmap. Training for a business requires specific data and validation.
RLCD and scoring rules
What it is
The work uses rewards based on distributions, including logarithmic and spherical components and an ordinal term for score. The idea is to reward useful probabilities, not only the winning label. The concrete implementation also includes approximations and details that should be read in the code.
Why learn
A training function with desirable properties does not guarantee calibration after finite training or domain change. Don’t present the math as a substitute for testing. The model sheets themselves report overconfidence and the need to adjust temperatures.
Learning a distribution and evaluating its quality are distinct steps; the empirical result can contradict the theoretical expectation.
✓ Apply with criteria
Not by itself. Capacity, optimization, sample, and usage distribution influence the result; calibration must be verified empirically.
✗ Avoid the automatic conclusion
Does choosing your own rule guarantee that 90% predicted is 90% real?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
probabilistic incentive
training signal
distance between levels
new data
Test your understanding
Does choosing your own rule guarantee that 90% predicted is 90% real?
Check the commented answer
Not by itself. Capacity, optimization, sample, and usage distribution influence the result; calibration must be verified empirically.
Gradual and reversible operation
What it is
A reasonable evolution starts in shadow mode: the model recommends, humans decide, and divergences are analyzed. Then an organization can approve simple, reversible actions within a narrow scope, keeping records and routing back to the manual queue. This is an evolution plan, not an automation already implemented.
Why learn
Define metrics and stopping criteria before the pilot. An increase in errors in a rare category may matter more than stability of the average. Keep schema versions, checkpoints, and policies together so you can reproduce and roll back changes when needed.
If a new product changes the meaning of billing, the distribution may change. Reevaluate with recent data before keeping the same threshold.
Key concepts
no action
data drift
version rollback
set threshold
Test your understanding
What to monitor besides average accuracy?
Check the commented answer
Errors by class, rejected inputs, unavailability, latency, review coverage, and changes in ticket distribution.
Final delivery: a process-explainable decision
What it is
Deliver the working project, the contract of the four questions, versioned dataset, inference report, and limitations analysis. Explain why the checkpoint was chosen, how the input is validated, and where human responsibility remains. The process should make it possible to understand the decision even without a text justification generated by the model.
Why learn
Compare three examples: one clear, one ambiguous, and one out of domain. For each, provide the expected label, received distribution, policy, and human conclusion. This delivery demonstrates mastery of the architecture without promising universal intelligence or hiding failures.
The evaluator should be able to install, re-run the command, find the result, and identify the next experiment needed.
Key concepts
repeat
find the source
known boundary
hypothesis
Test your understanding
What is the course completion criterion?
Check the commented answer
Can you run real weights, interpret choice/score/noul, evaluate a sample, distinguish a benchmark from a promise, and design an integration with permissions separate from the model?
Module summary
- An ambiguous operational contract or missing information. First define the desired outcome and provide the necessary facts.
- No. It delivers the adapted runtime, didactic evaluation, and an evolution roadmap. Training for a business requires specific data and validation.
- Errors by class, rejected inputs, unavailability, latency, review coverage, and changes in ticket distribution.
Select a snippet from the lesson to highlight or annotate. Questions and notes stay in your journey; export the JSON to back up.