Govern, measure and evolve|INEMA.CLUB-PRO

Lesson 5 · 20 min

LOOP-R: the process that learns

Hora handles nine hundred conversations a week and, the next week, works the same way. Every run generates evidence that nobody uses. LOOP-R closes the cycle: the run becomes evidence, the evidence becomes a hypothesis, the hypothesis becomes a test, and only what is proven goes into the next version. In this lesson you run a full cycle on Hora, from running to promoting.

Start the lesson
Four people standing in front of a whiteboard with arrows drawn in a circle and colorful sticky notes along the cycle.

In this lesson you will

  1. 1The seven steps of the cycle and who does each one
  2. 2Separate what is strong from what is noise in the evidence
  3. 3Write the hypothesis with a metric, guardrail and tolerance
  4. 4Test against the current version and decide with a five-line card

Goal: run a complete LOOP-R cycle, from executing to promoting, with a hypothesis, a test against the baseline and a decision from someone who did not propose it.

01The LOOP-R cycle

Seven steps and the loop back. Click each one to see what happens at Horizonte.

Click each block in the diagram to see what it does.

AI Management Area cycle (§05). The course and the LOOP-R videos summarize the same cycle in five letters, with nine assistants. Horizonte numbers: fictional, for study.

02The four decisions of the cycle

The cycle has seven steps, but four of them call for a manager's decision.

1. Strong, moderate or weak

Not every observation deserves a test. The LOOP-R ruler: with 90 cases or more, the evidence is strong and becomes a priority hypothesis; from 30 to 89, moderate; below 30, weak, and it goes to an "observed, not tested" list until more cases pile up.

A one-off complaint opens a question, not a change. That is the rule of the method: feedback does not change the agent in production.

At Horizonte640 delivery-time conversations: strong. "Tuesday sells more," with 14 orders: weak, goes to the list.

03Video: the method in five letters

Track 2 of the LOOP-R course, with Nei's avatar and voice. Watch it when you can: the lesson doesn't depend on it.

LOOP-R · Track 2: The method · 17 min 43 sVideo page with script →

04Templates to copy

The cycle log and the decision card. The tab next to it shows Hora's cycle filled in.

LOOP-R cycle log
LOOP-R CYCLE no. <0001> · process <...> · agent <name> · current version <vX.Y> · owner: <name>

1. RUN   version <vX.Y> from <date> to <date> · cases: <n>
2. OBSERVE   <what was counted, with N> · source: <log, sample>
3. CRITIQUE   what failed: <...> · strength: [ ] strong (N >= 90) [ ] moderate (30-89) [ ] weak (< 30)
              probable cause: <...>
4. PROPOSE     IF <change> THEN <metric> goes from <...> to <...> BECAUSE <...>
              WITHOUT WORSENING: <guardrail> (tolerance <...>, calculated as <...>) · critical = 0 · cost <ceiling>
              vetoed: <ideas and the invariant they violated>
5. TRY candidate <vX.Y+1> · 30 cases: <c/a/i/CRITICAL> · regressions <...>
              pilot: <how A and B are split> from <date> to <date> · minimum per variant: <n> (math: <...>)
6. VALIDATE    A: <result, N> · B: <result, N> · guardrails: <...>
              verdict: [ ] B won [ ] A stays [ ] insufficient sample
7. PROMOTE   proposed by <name> · decided by <someone else> on <date>: [ ] approve [ ] reject [ ] wait
              previous version kept at: <where> · rollback trigger: <lesson 6>

REPEAT: learning recorded at <where> · next hypothesis in the queue: <...>
Decision card (5 lines)
CYCLE <no.> · hypothesis: "<change in one sentence>"
Result:  <metric> <A> → <B>  (N = <nA> vs <nB>, minimum <n>)
Watched:    <guardrail 1> <A> → <B> (limit <...>) · critical <n> · 30 cases: regressions <n>
Cost:      <cost of the test and effect on cost per outcome>
Decision:    [approve]  [reject]  [wait for more data]  · decides: <name, not whoever proposed it> by <date>
Prompt · critique the hypothesis before the test
You are the critic of an improvement cycle. Below are the observed
evidence and the hypothesis proposed for an AI agent. Do not propose another
hypothesis and do not decide. Point out:

1. Whether the evidence has enough N to be strong, moderate or weak.
2. Whether the hypothesis states the change, the metric, the reason and what cannot get worse.
3. Whether the guardrail's tolerance was written beforehand and how it was calculated.
4. One alternative explanation for the problem that the test should rule out.

Evidence:
"""<paste>"""
Hypothesis:
"""<paste>"""
Simulation: illustrative example

Cycle no. 0001 · Hora v2.1 → v2.2 · Horizonte, fictional study company · relative weeks

  1. Run: v2.1, weeks 1 to 4, 3,600 conversations.
  2. Observe: 640 delivery-time or route conversations; Juliana took over 288 (45.0%). In the 30 cases, 2 of the 4 incorrect ones are about delivery time.
  3. Critique: strong strength (N = 640). Probable cause: the route and days-per-city table is missing.
  4. Propose: IF the table (owner: Valdir, logistics) goes into the package, THEN taken over from 45% to 25% or less. Without worsening: broken promises (tolerance 1.2 points), critical 0, AI up to R$ 0.35 per conversation. Vetoed: free shipping.
  5. Try: 30 cases on v2.2: 24 correct, 4 acceptable, 2 incorrect, 0 critical, no regression. Pilot in weeks 5 and 6, even days v2.1 and odd days v2.2; minimum of 89 per version.
  6. Validate: taken over A 148/330 (44.8%) × B 59/326 (18.1%); broken promises A 4/330 (1.2%) × B 5/326 (1.5%), limit 2.4%; critical 0 and 0. Verdict: B won.
  7. Promote: proposal by Patrícia (observation by Juliana); Carlos approves in week 7. v2.1 is kept.

05Practice

Hora's cycle with this lesson's numbers; your own cycle, on a process of yours.

1 · Hora's cycle 0001

30 min

Without looking at the example tab, fill in the cycle log with this lesson's Horizonte numbers, across the seven steps. Run the critic prompt on your hypothesis and answer the alternative explanation it points out. Finish with the five-line card.

Done when: the log has all 7 steps filled in; the hypothesis has a change, a metric with a starting value and a target, a reason, and a guardrail with a tolerance; the minimum per version is written with the math; the verdict is one of the three answers; and the card names who decides, someone other than who proposed it.

2 · A cycle on your own process

2 h in total

Use the lesson 4 panel to find the problem with strong or moderate evidence. Write the hypothesis with a guardrail, run the candidate on your 30 cases, run the pilot with the sample size that proves it, and take the card to whoever decides. If the sample doesn't reach the minimum, record "insufficient sample" and what to do next.

Done when: your cycle log has all 7 steps with numbers and N; the candidate passed the 30 cases with no critical; the verdict was given by someone who didn't propose; and the decision (approve, reject or wait) is recorded with a date.

06Quick test

5 questions. The explanation appears after your attempt.

Question 1 of 5

An important customer complained that Hora got the delivery time wrong. What does LOOP-R say to do?

07Summary

Seven stepsDo, observe, critique, propose, experiment, validate, promote. And repeat.
Strength of evidenceStrong at 90 or more; weak below 30 goes on the watch list.
Complete hypothesisChange, metric, why, and a guardrail with the tolerance written beforehand.
One of threeB won, A stays, or insufficient sample. Whoever did not propose it decides.

08Going further: why the company doesn't improve

Track 1 of the LOOP-R course (optional).

LOOP-R · Track 1: Why your company doesn't get better · 18 min 23 sVideo page with script →

09Going further: the loop's assistants

Track 3 of the LOOP-R course (optional): the nine assistants, each with its own role.

LOOP-R · Track 3: The loop's assistants · 17 min 48 sVideo page with script →

To go further

LOOP-R course: your company that learns on its own (5 tracks)LOOP-R: user guide and the worked exampleLOOP-R on video: the 5 tracks with chapters and captions
Previous lessonNext lesson