E1 · Your training
3 minRun curvas.py on the lesson 5 training and write down the term that paid the most at the end.
Done when: python3 kit/curvas.py $F/treino/ppo; echo "saída: $?" → treino: ppo · 20 iterações and saída: 0.
Lesson 6 · 16 min
Reading the training: curves, collapse, and reward hacking
A reward curve going up does not mean the robot learned to walk. In this lesson you read the log of your own training and of two real 150-iteration trainings, recorded for the course: in one of them the reward goes up because the robot learned to fall sooner. The other shows the fix and the limit of the fix.
Start the lesson
In this lesson you will
- 1Reward, episode length, and speed: the three numbers read together
- 2Reward hacking: it went from −4.31 to −1.12 while the episode dropped from 55 to 15 steps
- 3The fix with is_alive and its limit: 0.16 m/s
- 4The train → read → adjust → train again cycle
Goal: read the log of a training run and recognize collapse and reward hacking.
01The signals curvas.py looks for
Click each block. Each alert compares the first and the last quarter of the training.
Click each block in the diagram to see what it does.
Walking on its knees, shaking, and standing still are the classic shortcuts. All of them show up when you read these four numbers together, never the reward alone.
02The curves of your training
The progresso.csv from lesson 5.
Terminal · read your training
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
python3 kit/curvas.py $F/treino/ppo; echo "saída: $?"Example from the repository
treino: ppo · 20 iterações (médias do primeiro e do último quarto) recompensa inicial: -3.64 · final: -4.75 episódio médio: 46 → 58 passos (de 1000) velocidade para a frente no fim: 0.14 m/s · termo que mais pagou no fim: pose (1.00) alertas: 0 saída: 0
In 20 iterations the reward still oscillates and the term that pays the most is pose: the policy has barely left the default pose. No alerts, but nothing to learn from either. That is why the kit includes two larger training runs. Careful: curvas.py exits 0 even with an alert; read the alertas: line, not the exit code.
03Two real 150-iteration trainings
Generated with the same kit/treinar.py, 32 environments, seed 0. cai-cedo uses kit/recompensa.yaml (8 min); com-vivo adds the is_alive term with weight 5 (10 min). Only the progresso.csv is in the repository.
Excerpt · kit/recompensa-viva.yaml
is_alive: {weight: 5.0} # 5 × 0,02 s = +0,1 por passo de pé: ficar vivo passa a valer mais que a puniçãoTerminal · the two examples
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
for x in cai-cedo com-vivo; do python3 kit/curvas.py kit/exemplos/$x; echo; doneExample from the repository
treino: cai-cedo · 150 iterações (médias do primeiro e do último quarto) recompensa inicial: -4.31 · final: -1.12 episódio médio: 55 → 15 passos (de 1000) velocidade para a frente no fim: 0.15 m/s · termo que mais pagou no fim: upright (0.23) ALERTA: recompensa hackeada: a recompensa subiu porque o episódio encurtou; cada passo de pé custa mais do que paga, e cair cedo virou a melhor estratégia alertas: 1 treino: com-vivo · 150 iterações (médias do primeiro e do último quarto) recompensa inicial: 1.36 · final: 2.32 episódio médio: 57 → 63 passos (de 1000) velocidade para a frente no fim: 0.16 m/s · termo que mais pagou no fim: is_alive (6.27) alertas: 0
cai-cedo: the reward rose from −4.31 to −1.12 while the episode dropped from 55 to 15 steps. Since every step spent standing had a negative balance (the penalty for abrupt actions outweighed the rewards), falling early became the best strategy.
com-vivo: with is_alive weight 5 (+0.1 per step standing, 5 per second), the episode starts to grow (57 → 63) and no alert fires. But the term that pays the most is is_alive and the speed is 0.16 m/s: the training learned not to fall, not yet to walk.
04The tuning cycle
Run it, watch the curve reading stop the training, and run it again. This is the loop of this lesson.
to do
running
done
alert
Illustrative simulation in the browser. The alert in the curve reading is on purpose: it is the cai-cedo case.
05Practice
Read before you believe.
E2 · The shortcut
5 minRun both examples and write, in one sentence, why the reward in cai-cedo went up.
Done when: the cai-cedo output shows ALERTA: recompensa hackeada and alertas: 1; your sentence mentions the episode getting shorter (55 → 15 steps).
E3 · The next tweak
10 minThe com-vivo doesn't fall, but it walks at 0.16 m/s. Propose a tweak to one term (which one, old weight, new weight) and say what signal you would expect to see in the curves if it works.
Done when: you wrote one term from kit/recompensa.yaml with both weights and a measurable signal (e.g., final speed above 0.3 m/s without the episode getting shorter).
06Lesson quiz
The explanation appears after your answer.
Question 1 of 4
In cai-cedo, the reward went up and the episode dropped from 55 to 15 steps. What did the policy learn?skill
Correct
With a negative balance on every step spent standing, fewer steps = less penalty. curvas.py compares whole quarters of the training run precisely so it does not mistake this for noise.Not that one
With a negative balance on every step spent standing, fewer steps = less penalty. curvas.py compares whole quarters of the training run precisely so it does not mistake this for noise.Review: The curves of your trainingQuestion 2 of 4
The com-vivo triggers no alert. Why is it still not a good training run?
Correct
No alert is not the same as correct. A large is_alive removes the shortcut of falling, but it can create the shortcut of standing still; unitree_rl_lab uses only 0.15.Not that one
No alert is not the same as correct. A large is_alive removes the shortcut of falling, but it can create the shortcut of standing still; unitree_rl_lab uses only 0.15.Review: The curves of your trainingQuestion 3 of 4
Which reading detects "reward hacking" most reliably?
Correct
A single number always has a shortcut that fools it. Three signals and the dominant term together close the most common doors.Not that one
A single number always has a shortcut that fools it. Three signals and the dominant term together close the most common doors.Review: The signals curvas.py looks forQuestion 4 of 4
Why does the kit keep two example training runs instead of asking you to run 150 iterations?
Correct
You can repeat them (the commands are in kit/exemplos/LEIA.md). But the skill in this lesson is reading; the training time is optional.Not that one
You can repeat them (the commands are in kit/exemplos/LEIA.md). But the skill in this lesson is reading; the training time is optional.Review: The curves of your trainingQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Choose an option. The explanation appears after you try.
07In three sentences
ReadingReward, episode length, speed and dominant term, always together.
ShortcutIn cai-cedo, the reward went up because the robot learned to fall sooner.
CycleTrain, read, adjust one term and train again; with no alert it still isn't walking.