Lesson 6 · 18 min
Juliana's goal is still "conversations handled per day." But Hora handles most of them. Measuring Juliana by volume is measuring Hora, and it also pushes Juliana to take back the easy conversations. Her work has changed to deciding, reviewing, improving and taking care of people. In this lesson, the evaluation changes along with it.
Start the lesson
Goal: change the evaluation of a role from "how much it did" to "what it decided, reviewed, improved and taught," with evidence for each criterion.
Volume leaves the individual target; four criteria come in. Tap each one.
Criteria recommended by INEMA for roles that work with agents; they do not replace your company's evaluation policy or labor law.
Four steps, with Juliana's sheet as an example.
When the agent does most of the volume, the person's volume target measures the agent. Worse: it pushes the person to take the easy conversations back, or to leave the exception queue waiting.
Take volume out of the individual evaluation and keep it on the team dashboard. The evaluation question becomes what the person did with the work only they can do.
Juliana's old target: 120 conversations a day. With Hora, she handles the ones left over, harder and longer. The old target would fail her for the most important work.
Decide (exceptions on time, right verb, reason), review (errors found and errors that slipped through), improve (what became a rule or test), relationships and teaching (difficult customer, colleague trained).
Set weights by role. For people who serve customers, deciding and reviewing weigh more; for those who train the junior, teaching weighs more.
Decide 35% · Review 25% · Improve 20% · Relationships and teaching 20%.
For each criterion, write where the evidence comes from: the queue log, the review spreadsheet, the change requests, the cases told. An evaluation without a source becomes sympathy.
Describe each score on the 1-to-4 scale with a behavior. "3 = resolves on time with a reason, sometimes with the wrong verb" can be evaluated; "3 = good" cannot.
Queue log: 26 of 28 exceptions this month on time, all with a reason, 2 with the wrong verb (sent back what should have been stopped). Fictional numbers.
Never punish someone who points out an agent error: a found error is a contribution. Never use an agent's number as a personal target. Never compare a person's speed with the agent's.
In the evaluation conversation, the person brings three cases from the period: a hard decision, an error they caught and an improvement. The conversation starts from them, not from the spreadsheet.
Juliana brought the hotel request she held back, the deadline error from Hora that she logged and the test case she proposed. The conversation lasted 30 minutes and ended with a teaching goal: help Lucas in week 3.
The role's evaluation sheet and a prompt that looks for wrong incentives. The tab next to it shows Juliana's.
EVALUATION SHEET · role <...> · person <name> · period <...> · evaluator <name>
Agents the person works with: <names>
CRITERION WEIGHT WHAT COUNTS EVIDENCE (WHERE IT COMES FROM)
1 Decide <%> exceptions on time, right verb, <queue log>
written reason
2 Review <%> agent errors found and <review spreadsheet>
logged; errors that slipped through
3 Improve <%> cases that became a test, rule <change requests>
or process adjustment
4 Relationships <%> difficult customer resolved; <cases told; who was
and teaching colleague trained served or trained>
SCALE (describe each score per criterion)
1 = <behavior> · 2 = <...> · 3 = <...> · 4 = <...>
WHAT STAYS OUT: volume the agent handles · speed comparison with the agent
· agent error pointed out by the person (counts in their favor, under "review")
THE PERSON BRINGS 3 CASES: hard decision · error caught · improvement
SCORES: 1 <..> 2 <..> 3 <..> 4 <..> · agreement for the next period: <...>You review the evaluation sheet for a role that works alongside AI agents. Do not give scores and do not rewrite: point out risks. 1. Does any criterion, in practice, measure the agent's work and not the person's? 2. Could any criterion punish someone who points out an agent's mistake? 3. Does any criterion encourage the person to take back the easy work or to leave an exception waiting? 4. Does each criterion have evidence with a source? Does the scale describe behavior or just adjectives? Sheet: """<paste here>"""
Evaluation sheet · customer service · Juliana · Q4 · evaluator: Patrícia · works with Hora · Horizonte, a fictional study company
Not included: conversations per day (that's Hora's). Cases brought: the hotel request that was held; Hora's wrong deadline that was recorded; the test case "customer asks for a deadline the app doesn't show".
Redo Juliana's sheet; then one for a role on your team, with the person.
Using the matrix from lesson 2 and what Juliana now does, write her evaluation sheet with the four criteria, weights, evidence and scale. Run the wrong-incentive prompt and fix it.
Done when: the sheet has 4 criteria with weights adding up to 100%, evidence with a source for each, a 1-to-4 scale described by behavior in at least 2 criteria, "conversations per day" left out of the individual evaluation and mistakes found counted in her favor.
Choose a role on your team that already works with an agent, or will. Write the sheet and show it to the person before the next period starts. Note what they would change.
Done when: the sheet has the 4 criteria with evidence and scale, no criterion measures the agent's volume, the person read it before the period and at least one adjustment came from them.
4 questions. The explanation appears after your attempt.
Question 1 of 4
Hora handles most of the conversations. What does Juliana's "conversations per day" target measure?
Question 2 of 4
Juliana found and recorded 6 of Hora's mistakes this month. How does that count in her evaluation?
Question 3 of 4
Which scale can be evaluated?
Question 4 of 4
In the evaluation conversation, why does the person bring three cases?
Choose an answer. The explanation appears after you try.