E1 · Term by term
5 minRecord both rollouts and run termos.py on the walking rollout.
Done when: python3 kit/termos.py $F/treino/andando.npz; echo "saída: $?" → RECOMPENSA DO EPISÓDIO … 53.60 and saída: 0.
Lesson 4 · 16 min
Designing the gait reward
The policy learns what the reward pays for, not what you meant. In this lesson you read the gait terms of the mjlab velocity task, record one rollout walking and another standing still, measure how much each term paid, and find out that, with one weight swapped, standing still becomes worth more than walking.
Start the lesson
In this lesson you will
- 1The 7 gait terms the kit copies from mjlab, with name, weight and scale
- 2Walking: 53.60. Standing still with the same command: 42.32
- 3One swapped weight: standing still (40.22) beats walking (39.65)
- 4How unitree_rl_lab chose different terms for the same robot
Goal: measure how much each reward term pays in a recorded rollout and predict the effect of changing a weight before you train.
01Rewards and penalties
Click each block. Each term is worth weight × value × 0.02 s per policy step.
Click each block in the diagram to see what it does.
Names, weights, and std copied from the mjlab velocity task (commit 8bb2bc9, 10/18/2026). The kit uses a single std for pose.
02The kit's reward
A YAML file with one term per line. This is the file you will copy and change.
File · kit/recompensa.yaml
# Recompensa de marcha do kit (FTH 1). Nomes, pesos e std seguem a tarefa de velocidade do mjlab
# (src/mjlab/tasks/velocity/velocity_env_cfg.py, commit 8bb2bc9, 18/10/2026); "pose" usa um std só.
# Cada termo vale weight × valor × 0,02 s por passo da política. Positivo = prêmio; negativo = punição.
track_linear_velocity: {weight: 2.0, std: 0.5} # exp(-erro² / std²) entre comando e velocidade xy
track_angular_velocity: {weight: 2.0, std: 0.707} # o mesmo para a rotação em z
upright: {weight: 1.0, std: 0.447} # tronco em pé (gravidade sem componente x/y)
pose: {weight: 1.0, std: 0.3} # juntas perto da pose padrão
dof_pos_limits: {weight: -1.0} # quanto as juntas passaram do limite (rad)
action_rate_l2: {weight: -0.1} # ação que muda de um passo para o outro
foot_slip: {weight: -0.1} # pé que escorrega enquanto está no chão03Record two rollouts and measure each term
The finished policy walks for 10 s at 0.7 m/s and then stands still with command 0. termos.py redoes the math term by term; --comando-vx redoes the standing-still math as if the request were 0.7 m/s.
Terminal · record walking and standing still
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
mkdir -p $F/treino
python3 kit/andar_demo.py --vx 0.7 --gravar $F/treino/andando.npz | tail -1
python3 kit/andar_demo.py --vx 0 --gravar $F/treino/parado.npz | tail -1Terminal · terms of the walking rollout
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
python3 kit/termos.py $F/treino/andando.npz; echo "saída: $?"Terminal · standing still, charged as if it should walk
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
python3 kit/termos.py $F/treino/parado.npz --comando-vx 0.7Example from the repository
gravado: ~/fth/treino/andando.npz (500 passos) gravado: ~/fth/treino/parado.npz (500 passos)
rollout: andando.npz · 500 passos (10 s) · comando vx = 0.7 m/s termo weight valor médio total track_linear_velocity 2 0.930 18.59 track_angular_velocity 2 0.814 16.28 upright 1 0.995 9.95 pose 1 0.925 9.25 dof_pos_limits -1 0.000 0.00 action_rate_l2 -0.1 0.421 -0.42 foot_slip -0.1 0.054 -0.05 RECOMPENSA DO EPISÓDIO 53.60 saída: 0
While walking, the biggest payers are tracking velocity (18.59) and turning (16.28). The penalties barely weigh: the policy is already smooth.
rollout: parado.npz · 500 passos (10 s) · comando vx = 0.7 m/s (trocado na conta) termo weight valor médio total track_linear_velocity 2 0.140 2.80 track_angular_velocity 2 0.999 19.98 upright 1 0.999 9.99 pose 1 0.956 9.56 dof_pos_limits -1 0.000 0.00 action_rate_l2 -0.1 0.003 -0.00 foot_slip -0.1 0.000 -0.00 RECOMPENSA DO EPISÓDIO 42.32
Standing still with a 0.7 m/s request, linear tracking drops from 18.59 to 2.80, but turning, upright, and pose pay even more. The total is 42.32 versus 53.60: standing still already earns 79% of walking.
ERRO: não achei ~/fth/treino/nao-existe.npz. Grave antes com: python3 kit/andar_demo.py --gravar ~/fth/treino/nao-existe.npz saída: 1
termos.py reads a .npz file recorded by andar_demo.py. Record first; the error tells you the command.
04Change one weight and redo the math
A copy of the YAML with track_linear_velocity at 0.5 instead of 2.0. Nothing is trained: only the math changes.
Terminal · weight 0.5 on tracking
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
sed "s/track_linear_velocity: {weight: 2.0/track_linear_velocity: {weight: 0.5/" kit/recompensa.yaml > $F/treino/minha.yaml
python3 kit/termos.py $F/treino/andando.npz --recompensa $F/treino/minha.yaml | tail -1
python3 kit/termos.py $F/treino/parado.npz --comando-vx 0.7 --recompensa $F/treino/minha.yaml | tail -1Example from the repository
RECOMPENSA DO EPISÓDIO 39.65 RECOMPENSA DO EPISÓDIO 40.22
First line: walking, 39.65. Second: standing still, 40.22. With this weight, a training run that finds "stand still and upright" is rewarded for it: that is reward hacking before any training. Doing this math takes seconds; finding out after a training run costs hours of GPU.
05Same robot, different recipe
unitree_rl_lab (commit 4960b84) trains the same G1 with different terms. Click to compare.
1. Alive
alive with weight 0.15: a small reward for each step spent upright. Lesson 6 shows what happens when it is missing and when it is too large.
2. Height
base_height with weight −10 and a target of 0.78 m: penalizes the pelvis being off height. It keeps the robot from learning to walk crouched or on its knees.
3. Tilt
flat_orientation_l2 with weight −5 penalizes a tilted torso, the role upright plays in mjlab.
4. Gait
gait with weight 0.5 rewards the stepping pattern: a period of 0.8 s, with the two feet half a period out of phase (alternating). Without a term like this, the policy may find a strange way to get around, like dragging its feet.
06Practice
Check the reward before spending a training run.
E2 · Standing still
5 minRun the standing-still rollout charged as if it should walk, and work out how much it is worth compared to walking.
Done when: the output ends in 42.32 and you wrote "standing still is worth 79% of walking".
E3 · The weight that flips the game
10 minRun the block with a weight of 0.5. Then, in your own copy of the YAML, find the lowest track_linear_velocity weight at which walking is still worth more than standing still.
Done when: with 0.5, the second line (40.22) is greater than the first (39.65); you noted the weight at which walking starts winning again (on 10/19/2026: 0.55 still loses, 0.6 wins).
07Lesson quiz
The explanation appears after your answer.
Question 1 of 4
Standing still with a command of 0.7 m/s, the robot still adds up to 42.32. Where does most of it come from?skill
Correct
A turn command of 0 is "met" by a robot that doesn't turn, and upright and pose reward posture. All of them pay without walking: that's why the linear tracking has to carry weight.Not that one
A turn command of 0 is "met" by a robot that doesn't turn, and upright and pose reward posture. All of them pay without walking: that's why the linear tracking has to carry weight.Review: The kit's rewardQuestion 2 of 4
With track_linear_velocity at 0.5, standing still is worth 40.22 and walking 39.65. What is a training run likely to learn?
Correct
The algorithm maximizes the sum. If standing still pays more, that's what it figures out. Doing the math on a recorded rollout shows you the risk before training.Not that one
The algorithm maximizes the sum. If standing still pays more, that's what it figures out. Doing the math on a recorded rollout shows you the risk before training.Review: The kit's rewardQuestion 3 of 4
What is action_rate_l2 for?
Correct
Jitter costs little in the simulator; on the robot it heats up the motors and drains the battery. The term exists so the gait comes out smooth.Not that one
Jitter costs little in the simulator; on the robot it heats up the motors and drains the battery. The term exists so the gait comes out smooth.Review: Rewards and penaltiesQuestion 4 of 4
Why does unitree_rl_lab have base_height with a weight of −10?
Correct
Each term closes a door to a shortcut. Walking on its knees can still track velocity; the height term removes that path.Not that one
Each term closes a door to a shortcut. Walking on its knees can still track velocity; the height term removes that path.Review: Same robot, different recipeQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Choose an option. The explanation appears after you try.
08In three sentences
SumThe reward is a sum of weighted terms; each term pays or punishes at every step.
MeasureWalking 53.60, standing still 42.32; with tracking at 0.5, standing still starts to win.
HabitBefore training, measure the reward of a good rollout and a bad one.