The Hora dashboard shows requests going up, and Carlos concludes it is paying for itself. The dashboard only measures volume: it doesn't tell you whether Hora got it right, how much each request cost, or how much human work is still needed. In this lesson you replace that dashboard with five agent indicators, each with a numerator, a denominator, a baseline and a limit that triggers an action.
1What is missing in each column of a volume dashboard
2The agent's 5 indicators: quality, time, cost, autonomy and intervention
3Baseline when the "before" was never measured
4Limits that trigger action, and the indicator that improves when the agent gets worse
Goal: track the agent on a 5-indicator dashboard with numerator, denominator, baseline and limit, fixing the dashboard that only shows volume.
01 · Five indicators and one alarm
Each indicator is a fraction with a period and a source. The critical error stays out of all of them: it is counted separately. Click each one.
Click each block in the diagram to see what it does.
Indicators from the Gestoria method and the AI Management Area (§06). In the Gestoria tool, the Overview panels show strict accuracy, critical errors, autonomous work, and cost per case, calculated over the lab's cases. Horizonte values: fictional, for study.
02 · From the volume dashboard to the agent dashboard
The lab's F11 was already fixed in G2 with three process indicators. Here the topic is the agent.
1. What each column does not show
The lab dashboard has four columns. Conversations is volume: it goes up with a promotion, the time of year, or too much discount. Orders closed has no before and no comparison. Average satisfaction comes from a button that about 6% of customers answer.
Response time measures the speed of the first message, not of the order. No column tells you whether Hora got it right, how much it cost, or how much human work it created.
Test for each columnDoes this number change if Hora gives a wrong discount? If it doesn't, it's no good for governing the agent.
2. Baseline without the before
Data collection started on the same day Hora came in, in July. The "before Hora" was never measured and can't be measured now. What you can do: measure four weeks of the current version and use that as the baseline against which every change will be compared.
Estimates of the before (like the 8 to 12 minutes per order from G4) remain estimates and show up that way on the dashboard, never as observed.
At HorizonteBaseline: Hora v2.1, four weeks, 3,600 conversations and 680 orders completed. Every dashboard value states the period and the n.
3. Limit that triggers action
Each indicator gets a limit and a defined action for when it is crossed. Quality below 90% in a week: the sample doubles and the cause goes into the weekly review. One critical error: the action goes back to yellow right away and an incident is opened.
The limits are a decision for whoever is accountable for the operation, looking at the baseline. Write them before you need them.
At Horizontep95 of time above 6 h: look at the approval queue. Cost above R$ 7.50 in the month: review. Intervention above 6 min per order: see where the hours went.
4. The indicator that misleads
If the target is "raise autonomy", the fastest way is for Hora to stop asking for approval. The number gets better and the operation gets worse. That's why autonomy and intervention are only read next to quality and critical errors.
In the Work Trend Index 2025, Microsoft proposed the human-agent ratio metric: the balance between human oversight and agent efficiency. It's a vendor proposal; your dashboard measures that balance with indicators 4 and 5.
Question for each indicatorHow does this number improve without the operation improving? If the answer is easy, it needs a pair that watches over it.
03 · Templates to copy
The one-page dashboard and the sheet for each indicator. The tab next to this one shows Hora's.
Agent dashboard (5 indicators)
AGENT DASHBOARD · <name> v<...> · period <from> to <to> · dashboard owner: <name>
Baseline: <version>, <n weeks>, <cases> cases · data source: <systems>
| # | Indicator | Numerator ÷ denominator | Baseline (n) | This week | Limit → action | Owner |
|---|---|---|---|---|---|---|
| 1 | Quality | correct ÷ judged (sample of <n>/week) | <%> (<n>) | <%> | < <%>: <action> | <name> |
| | Critical errors | count, outside any average | <n> | <n> | 1: <immediate action> | <name> |
| 2 | Time | median and p95 of <end − start>, completed cases | <...> | <...> | p95 > <...>: <action> | <name> |
| 3 | Cost per outcome | total process cost ÷ completed cases | R$ <...> | R$ <...> | > R$ <...>: <action> | <name> |
| 4 | Autonomy | completed without intervention ÷ completed | <%> | <%> | only with 1 and criticals stable | <name> |
| 5 | Human intervention | human minutes ÷ completed cases | <min> | <min> | > <min>: <action> | <name> |
OFF THE DASHBOARD (volume, not outcome): <conversations, messages, satisfaction with low response>
ESTIMATED, NOT OBSERVED: <"before" values, with the source of the estimate>
Indicator sheet
INDICATOR <n> · <name> · agent <name> · owner: <name>
Management question: <what we want to know>
Numerator: <what is counted> · Denominator: <the universe>
Period: <week> · compare with: <baseline, same type of period>
Source: <system, field, sample> · who pulls it: <name>
Baseline: <value> · [ ] observed (n = <...>, from <date> to <date>) [ ] estimated (source: <...>)
Limit: <value> → action: <what happens, who does it, how fast>
Pair that watches: <indicator that exposes this one if it improves the wrong way>
Do not confuse with: <what this number does not show>
Prompt · how this indicator misleads
You are an indicator analyst. Below is the dashboard of an AI agent
with 5 indicators. Do not suggest new indicators. For each one:
1. Say whether the denominator is clear and whether it can change on its own and mislead
the comparison.
2. Give a way for the number to improve without the operation improving.
3. Say which other indicator on the dashboard would expose that way.
4. Point out whether any critical error might be hidden inside an average.
Dashboard:
"""<paste>"""
Simulation: illustrative example
Hora v2.1 · 4-week baseline · 3,600 conversations, 680 orders completed · Horizonte, fictional study company
Indicator
Fraction
Baseline
Limit → action
1 Quality
correct ÷ judged (40 drawn per week)
148 ÷ 160 = 92.5%
< 90%: sample doubles
Critical errors
count
0
1: action goes back to yellow + incident
2 Time
median and p95, contact → order confirmed
18 min · 4 h 40 min
p95 > 6 h: approval queue
3 Cost
process cost ÷ orders completed
4,542 ÷ 680 = R$ 6.68
> R$ 7.50: review
4 Autonomy
without intervention ÷ completed
510 ÷ 680 = 75.0%
no target on its own
5 Intervention
human minutes ÷ completed
2,988 ÷ 680 = 4.4 min
> 6 min: see where
Off the dashboard: conversations (volume) and average satisfaction (about 6% respond). Estimated: human time before Hora (8 to 12 min per order, G4).
Column
What it measures
What it does not show
Conversations
volume of conversations
whether the conversation became the right order
Orders closed
orders per week
the before, the discount given, the orders corrected
Average satisfaction
stars from about 6% of customers
the opinion of the other 94%
Response time
seconds until the 1st message
how long the order took to be confirmed
Data from the lab's indicadores.json (fictional): from 702 conversations and 141 orders (week of 09/01) to 901 and 168 (week of 10/13).
Indicator without a denominator. "12 conversations taken over" says nothing without the total.
Critical error in the average. The error that costs money disappears inside 99%.
Average time. One stuck order pulls the average; use median and p95.
Autonomy as a target. It goes up when the agent stops asking for help.
Estimate as observed. The before that nobody measured shows up as estimated, always.
04 · Practice
Hora's dashboard for practice; your agent's for real.
1 · Hora's dashboard
30 min
Open the lab panel and write, for each of the 4 columns, what it measures and what it does not show. Then build Hora's panel from the template, using the baseline numbers from this lesson, and fill in the indicator sheet for number 4 (autonomy) with the pair that watches it.
Done when: the 4 columns of the lab panel have "measures" and "does not show"; the panel has the 5 indicators with numerator, denominator, baseline with n, limit with action, and owner; critical errors appear outside any percentage; and the autonomy sheet says which indicator watches it.
2 · Your agent's panel
45 min
Build your agent's panel with the 5 indicators. Write down where each number comes from and who pulls it. Start the 4-week baseline collection, changing nothing in the agent during that period, and mark everything from before as estimated.
Done when: the panel is saved with the 5 sheets; each current value is marked as observed (with n and period) or estimated (with the source); the date the 4 weeks end is on the calendar; and the prompt "how this indicator misleads" has been run and answered.
The quarter's goal is to raise Hora's autonomy from 75% to 90%. What is the risk?
Correct
It's the indicator that improves when the agent gets worse. It needs a pair that watches it: quality in the sample and critical errors.
Not this one
It's the indicator that improves when the agent gets worse. It needs a pair that watches it: quality in the sample and critical errors.Review: Five indicators and one alarm
Question 3 of 5
Which of these is an honest baseline for Hora?
Correct
You don't invent a past that was never measured. The baseline becomes the measured present, against which every change will be compared.
Why use the median and p95 for time, instead of the average?
Correct
The median shows the typical case, and p95 keeps the stuck ones from disappearing. Requests waiting for approval show up in p95.
Not this one
The median shows the typical case, and p95 keeps the stuck ones from disappearing. Requests waiting for approval show up in p95.Review: Five indicators and one alarm
Question 5 of 5
This week's sample had 39 correct out of 40 and one critical error. What does the panel look like?
Correct
The critical error is counted separately and has its own action: the status goes back to yellow and opens an incident. A high percentage does not cancel the alarm.
Not this one
The critical error is counted separately and has its own action: the status goes back to yellow and opens an incident. A high percentage does not cancel the alarm.Review: Templates to copy
Quiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.
Choose an answer. The explanation appears after you try.
06 · Summary
Five fractionsQuality, time, cost, autonomy, and intervention, each with a denominator.
Critical errors apartCounted outside the averages, with its own action.
BaselineFour weeks measured now; the earlier period only as estimated.
Limit and pairEvery indicator has a limit with an action and another indicator that watches it.
