1 · One week of logging
5 min a dayFor the function you chose, log every run in EXECUCOES.md with the three minimum fields.
Done when: the file has at least 10 rows and you can state the week's accuracy rate (e.g., 8 out of 10).
Lesson 6 · 17 min
Closing the loop: failure log and observability
What isn't observed isn't cultivated. This lesson follows a clinic front-desk agent from its first error to supervised autonomy, and shows the minimum you need to record to replace "I think it got better" with "it improved from 88% to 95%".
Start the lesson
In this lesson you will
- 1How an error becomes a permanent rule
- 2Comparing agent versions with numbers
- 3Minimum observability: three fields
- 4Finding the error pattern worth fixing first
Goal: record each run with the minimum set of fields and turn a real error into a new rule, comparing the result before and after.
01The clinic case, in six stages
Scroll to follow along. The numbers are from the case described in the project.
Scroll the page: each step below lights up its part of the diagram.
- 1/7
Starting environment
Doctors and specialties, each doctor's hours, procedures and duration, clinic rules, and access to the schedule. The agent only suggests; the receptionist approves each appointment.
- 2/7
First failure
The agent books 7:30 PM with a doctor who doesn't see patients after 6 PM. The schedule showed "free," but a free schedule slot is not an available doctor. The failure is logged with date, decision, result, and root cause.
- 3/7
New rule
"Before suggesting, check the schedule AND the doctor's working hours. Outside working hours, never, even if the schedule shows free." Takes effect from the next run onward.
- 4/7
Evaluation
Of the next 20 interactions, 18 were correct. The 2 failures become two more rules: the procedure's duration must fit in the time slot, and 15 minutes between appointments.
- 5/7
Stabilization
40 interactions in two weeks: 20/20 in each, zero corrections from the receptionist, response time from 4 minutes to 45 seconds.
- 6/7
Autonomy
With the history, the agent starts confirming on its own. The human gets a daily report and reviews only what the agent flagged as an exception: cutoff time, new patient, long procedure.
- 7/7
After 2 months
Rules: from 5 generic ones to 14 born from real failures. 320 runs logged, 7 failures documented, 97.8% accuracy, version v1.0 → v1.4. The model is the same as on day one; if the clinic switches models, the garden stays.
02The difference between two versions
What observability shows (project example, 100 runs in each version).
1. v1.0 · 88%
Three error patterns: outside the professional's hours (5), time slot too short for the procedure (4), and an ambiguous reply about cancellation (3).
Each pattern produced a new rule: four rules modified.
2. v1.1 · 95%
Patterns 1 and 2 disappeared. Pattern 3 dropped from 3 to 1 occurrence.
A new pattern showed up: two simultaneous procedures requesting the same exam room. The cycle goes on.
The investment
From 88% to 95% with rules and fixes. Nobody switched the model: they looked at the data and adjusted the environment.
3. What you can decide
Compare versions on the same scenarios. Find patterns ("70% of errors are long procedures"). Measure the cost of an error ("each rescheduling costs R$ 150"). Decide where to invest first.
03Run log
Start with the three fields. The full version comes when the volume calls for it.
EXECUCOES.md · minimum viable
| date | what was asked | what was delivered | did it work? (yes / no / partly) | |---|---|---|---| | | | | |
EXECUCOES.md · full
| date | agent version | goal | decision made | tool used | time | cost | result | error | human intervention | rating (correct / partial / wrong) | fix applied | |---|---|---|---|---|---|---|---|---|---|---|---| | | | | | | | | | | | | |
From failure to rule (log template)
## Failure <date> What happened: Decision the agent made: Root cause (missing context, rule, example, or tool?): Smallest possible fix: New rule (exact text that went into the file): Agent version: v1.0 → v1.1 How I'll know it worked: next 20 runs, count the hits
04Practice
One week of observation.
2 · A rule born from an error
20 minTake the failure that repeated most during the week and fill in the "from failure to rule" template. Put the rule in the file and bump the version.
Done when: the rule is written in the agent's file, the version went up (v1.0 → v1.1), and the next runs are being counted so you can compare.
05Quick quiz
4 questions.
Question 1 of 4
The calendar showed the time slot as free and the agent booked outside the doctor's working hours. What was the root cause?
Correct
A free calendar slot is not an available doctor. The fix was a permanent rule in the environment, not a new model.Not this one
A free calendar slot is not an available doctor. The fix was a permanent rule in the environment, not a new model.Review: The clinic case, in six stagesQuestion 2 of 4
What is the minimum viable observability?
Correct
Three fields are enough to calculate accuracy and close the first cycle. More fields come in when the volume calls for it.Not this one
Three fields are enough to calculate accuracy and close the first cycle. More fields come in when the volume calls for it.Review: Run logQuestion 3 of 4
In v1.1 a new error pattern appeared (exam room conflict). That means that...
Correct
The old patterns are gone and the rate went up from 88% to 95%. New patterns show up when the old ones get out of the way.Not this one
The old patterns are gone and the rate went up from 88% to 95%. New patterns show up when the old ones get out of the way.Review: The difference between two versionsQuestion 4 of 4
After 2 months, what does the clinic have that a competitor can't copy by hiring the same model?skill
Correct
The garden belongs to the clinic. If it switches models, the garden stays; whoever copies the model starts from zero.Not this one
The garden belongs to the clinic. If it switches models, the garden stays; whoever copies the model starts from zero.Review: The clinic case, in six stagesQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Choose an option. The explanation appears after your attempt.
06Summary
Error becomes ruleA log with the root cause, the smallest fix, the rule in the file, and a new version.
Compare versionsSame scenarios, before and after: 88% → 95% without changing the model.
Start with three fieldsRequest, delivery, did it work. That already closes the first cycle.