Router, context, and processing
Avoid routing an input to the wrong model.
- The Router chooses before predicting
- Portuguese needs explicit choice in the lab
- Token budget is shared
- Many classes compete for the same space
- Resident Model and language switching
- Batching and perceived latency
The Router chooses before predicting
What it is
Router.route can select a checkpoint without running the neural network. The decision uses explicit model or task indication, optional workflow configuration, the provided language, and detection of writing or language. This avoids relying only on later confidence as a defense against a model that doesn’t understand the text.
Why learn
The current code should prevail over simplified diagrams. typed-decisions is not selected automatically by default. Enabling auto_task_detection does exact matching of the identifiers of four workflows; that’s not a universal understanding of the domain.
Router().route({"message":"Duplicate charge"}, questions, lang="pt") selects multilingual.
✓ Apply with criteria
No. Optional detection requires the exact set of identifiers for a workflow, and it comes disabled by default.
✗ Avoid the automatic conclusion
Is an urgency field enough to select typed-decisions?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
pre-selection
explicit indication
writing system
specialized scheme
Test your understanding
Is an urgency field enough to select typed-decisions?
Check the commented answer
No. Optional detection requires the exact set of identifiers for a workflow, and it comes disabled by default.
Portuguese needs explicit choice in the lab
What it is
Portuguese uses the Latin alphabet, like English, and very short messages provide little evidence for automatic detection. Since this lab’s contract already states Portuguese support, the implementation fixes multilingual. The SDK remains available for anyone who wants to try dynamic routing in another flow.
Why learn
A simple rule based on reliable product information can beat a language heuristic. If the interface knows the queue is Brazilian, it doesn’t need to guess that in every ticket. Test abbreviations, typos, and mixed messages separately.
Hello has little information; the support queue defines the checkpoint, not the confidence of a later prediction.
-
1
Observe
Hello has little information; the support queue defines the checkpoint, not the confidence of a later prediction.
-
2
Define
Portuguese uses the Latin alphabet, like English, and very short messages provide little evidence for automatic detection.
-
3
Check
No. Declared language coverage doesn’t guarantee performance in vocabulary, in your products, and in the expressions used by your customers.
Key concepts
reliable metadata
little evidence
approximation
scope
Test your understanding
Does the multilingual model remove the need to test in Portuguese?
Check the commented answer
No. Declared language coverage doesn’t guarantee performance in vocabulary, in your products, and in the expressions used by your customers.
Token budget is shared
What it is
The internal prompt reserves space for the question and options; the rest accommodates the state. The multilingual checkpoint typically uses max_len 1024 and head_max_len 256. These numbers don’t equal characters, nor do they ensure that any document with fewer than 1024 words will fit.
Why learn
The SDK can truncate the state. In the adaptation, we calculate the space available for each question and reject inputs that would lose tokens. Cutting the end of a message can remove the very negation, the date, or the main request.
A long history ends with I already received the refund. If that part is cut, the decision may represent a situation that has already been resolved.
Reference command / schema
from laya import Router
from practical.engine import QUESTIONS
router = Router(max_loaded=1)
router.preload(["multilingual"])
result = router.predict(
{"message": "Fui cobrado duas vezes."},
QUESTIONS, model="multilingual", lang="pt")
Key concepts
tokenizer unit
question and options
ticket content
input loss
Test your understanding
Why isn’t the interface limit in characters enough?
Check the commented answer
The relationship between characters and tokens varies. The definitive protection uses the real tokenizer and the smallest available space across the four questions.
Many classes compete for the same space
What it is
Each alternative needs markers and enough text to remain distinguishable. Dozens of options consume the header budget. In large classifications, similar labels can end up almost identical after cuts. The project report uses Banking77 to show this limit.
Why learn
One alternative is hierarchy: first choose a family, then a subcategory. This reduces options per step, but errors from the first choice propagate. Compare the full set and keep a review path for cases between families.
First financial vs. technical; then duplicate charge vs. invoice, within the chosen family.
✓ Apply with criteria
No. It can reduce the state space and increase cost; you still need to measure quality, memory, and latency with the new settings.
✗ Avoid the automatic conclusion
Does increasing head_max_len solve any number of categories?
Don’t accept an answer just by the field name or by the appearance of precision. Check the definition and the context of this section.
Key concepts
number of classes
selection stages
chained error
discriminative signal
Test your understanding
Does increasing head_max_len solve any number of categories?
Check the commented answer
No. It can reduce the state space and increase cost; you still need to measure quality, memory, and latency with the new settings.
Resident Model and language switching
What it is
The default Router keeps a few hot models and can drop the least recent. When two languages switch checkpoints and only one can fit in the cache policy, reconstruction cost dominates the call. Preload or a higher max_loaded resolves the switching, as long as memory can hold the models.
Why learn
Measurements must distinguish the hot path from the first load and from switching. In practice, the app avoids switching by keeping only multilingual. The lock prevents two requests from modifying loading and the shared runtime at the same time.
An API that recreates Router for every ticket loses the benefit of residence. Create an instance per process.
Key concepts
eviction by use
loaded model
first load
serialized access
Test your understanding
Do more workers always increase capacity?
Check the commented answer
No. Each process can load its own copy of the weights. Measure memory and throughput before increasing the number of workers.
Batching and perceived latency
What it is
Laya groups the questions from a state into a single call. The lab does not implement aggregation of multiple tickets from different users. The latency seen in the interface includes waiting, HTTP, initial loading, and inference; runtime.inference_ms reports only the step measured by the engine after preparation.
Why learn
Comparing milliseconds without stating measurement boundaries leads to fragile conclusions. Record the number of questions and the length distribution of the inputs. For operations, high percentiles and failures matter as much as the average; a small sample doesn’t estimate the tail well.
Four questions in a ticket aren’t four customers served. Question throughput and ticket throughput have different denominators.
Key concepts
batch of questions
median
latency tail
units per second
Test your understanding
Is it correct to divide any latency by four and promise that time per ticket?
Check the commented answer
No. Dividing can estimate average per-question cost within that batch; the user expects completion of the whole ticket.
Module summary
- No. Optional detection requires the exact set of identifiers for a workflow, and it comes disabled by default.
- The relationship between characters and tokens varies. The definitive protection uses the real tokenizer and the smallest available space across the four questions.
- No. Each process can load its own copy of the weights. Measure memory and throughput before increasing the number of workers.
Select a snippet from the lesson to highlight or annotate. Questions and notes stay in your journey; export the JSON to back up.