E1 · Short training
5 minRun the 20-iteration training block.
Done when: timeout 900 python3 kit/treinar.py --iters 20; echo "saída: $?" → checkpoint salvo: ~/fth/treino/ppo/modelo_20.pt, saída: 0, arquivo existe: modelo_20.pt and 21.
Lesson 5 · 16 min
PPO with thousands of robots in parallel
The algorithm that trains almost every humanoid gait today is PPO, running thousands of copies of the robot on the GPU. In this lesson you learn the collect, estimate, and update loop, run a ~120-line PPO with mjlab's values on CPU, with 16 robots, and see the command that does the same with 4,096.
Start the lesson
In this lesson you will
- 1The loop: collect 24 steps per robot, estimate the advantage, update the network
- 2The G1's hyperparameters in mjlab: lr 1e-3 adaptive, clip 0.2, 5 epochs
- 320 iterations on CPU: checkpoint and progresso.csv
- 4At scale (shown): 4,096 environments on the GPU, 30,000 iterations
Goal: run a short PPO training on CPU that saves a checkpoint and a log, and read the values from a real training run in mjlab.
01One PPO iteration
Click each block. Training repeats this loop thousands of times.
Click each block in the diagram to see what it does.
Actor and critic network: 512-256-128 with ELU, normalized observations, as in mjlab for the G1.
02The values, copied from mjlab
kit/ppo.py uses the names and defaults from the G1's rl_cfg.py in mjlab (commit 8bb2bc9, 10/18/2026; rsl-rl-lib library 5.5.1).
Excerpt · kit/ppo.py
PADRAO = dict(clip_param=0.2, entropy_coef=0.01, value_loss_coef=1.0, num_learning_epochs=5,
num_mini_batches=4, learning_rate=1.0e-3, desired_kl=0.01, gamma=0.99, lam=0.95,
max_grad_norm=1.0, num_steps_per_env=24, init_std=1.0, hidden_dims=(512, 256, 128))03Train 20 iterations on CPU
16 G1s in parallel in MuJoCo, one core, with the reward from lesson 4. The 900 s cap is a safety margin: here it took seconds.
Terminal · short training run
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
timeout 900 python3 kit/treinar.py --iters 20; echo "saída: $?"
test -f $F/treino/ppo/modelo_20.pt && echo "arquivo existe: modelo_20.pt"
wc -l < $F/treino/ppo/progresso.csvExample from the repository
PPO em CPU: 16 ambientes × 24 passos = 384 amostras por iteração · obs 96 → ação 29 it 1/20 · recompensa média -1.75 · episódio médio 24.0 passos · quedas 0 it 2/20 · recompensa média -3.33 · episódio médio 42.8 passos · quedas 5 it 3/20 · recompensa média -4.85 · episódio médio 58.6 passos · quedas 10 it 4/20 · recompensa média -3.90 · episódio médio 50.2 passos · quedas 4 it 5/20 · recompensa média -4.38 · episódio médio 52.6 passos · quedas 7 it 6/20 · recompensa média -4.38 · episódio médio 56.5 passos · quedas 4 it 7/20 · recompensa média -5.37 · episódio médio 69.7 passos · quedas 9 it 8/20 · recompensa média -5.28 · episódio médio 68.0 passos · quedas 4 it 9/20 · recompensa média -4.68 · episódio médio 56.8 passos · quedas 9 it 10/20 · recompensa média -4.73 · episódio médio 61.6 passos · quedas 7 it 11/20 · recompensa média -4.45 · episódio médio 52.6 passos · quedas 5 it 12/20 · recompensa média -4.68 · episódio médio 56.9 passos · quedas 9 it 13/20 · recompensa média -4.22 · episódio médio 52.6 passos · quedas 7 it 14/20 · recompensa média -4.56 · episódio médio 55.0 passos · quedas 5 it 15/20 · recompensa média -4.81 · episódio médio 57.1 passos · quedas 8 it 16/20 · recompensa média -4.88 · episódio médio 59.0 passos · quedas 4 it 17/20 · recompensa média -4.56 · episódio médio 56.6 passos · quedas 9 it 18/20 · recompensa média -5.19 · episódio médio 63.4 passos · quedas 7 it 19/20 · recompensa média -4.47 · episódio médio 51.8 passos · quedas 6 it 20/20 · recompensa média -4.64 · episódio médio 57.5 passos · quedas 8 checkpoint salvo: ~/fth/treino/ppo/modelo_20.pt log: ~/fth/treino/ppo/progresso.csv (20 linhas) · tempo total: 9 s saída: 0 arquivo existe: modelo_20.pt 21
Check the format, not the value: 20 it lines, checkpoint salvo:, the file exists, and progresso.csv has 21 lines (header + 20). The negative average reward and the falls show that 20 iterations don't teach the G1 to walk; mjlab uses max_iterations=30_000. The total time changes from one machine to another.
ModuleNotFoundError: No module named 'mujoco'
The system Python doesn't have MuJoCo. Activate the bench's venv (. ~/fth/.venv/bin/activate), which is what the first line of each block does.
04The same training at scale (shown, not run here)
Commands from the official READMEs. They need an NVIDIA GPU and a separate environment for each tool: don't mix them into the course's venv.
Shown · mjlab (README, commit 8bb2bc9)
uv run train Mjlab-Velocity-Flat-Unitree-G1 --env.scene.num-envs 4096
Shown · unitree_rl_lab (README, Isaac Lab)
python scripts/rsl_rl/train.py --headless --task Unitree-G1-29dof-Velocity
05Where real training runs
Three current paths, all on the GPU. Click each one.
1. mjlab
Isaac Lab's "manager-based" API on top of MuJoCo Warp 3.16, Apache-2.0, with uv. It requires an NVIDIA GPU (on macOS, evaluation only). It is the closest relative of this kit: the same term names.
2. Isaac Lab
unitree_rl_lab, BeyondMimic and Berkeley Humanoid Lite train on Isaac Lab 2.x (pin the version each README indicates). Isaac Lab 3.0 is in early access (EA, 09/16/2026), requires Isaac Sim 6.1 and a new Python 3.12 environment with uv.
(reported) According to the Isaac Lab benchmark table cited in robotica-real2sim, Isaac Lab's own G1 rough-terrain velocity task, with 4,096 environments, runs about 82 thousand steps/s on an RTX 4090 in 3.0 beta; the number varies with the version.
3. Playground
MuJoCo Playground has the G1JoystickFlatTerrain task, with Python 3.12 + uv, on JAX/MJX; training with MJWarp was announced in beta.
4. Legacy
legged_gym, on top of Isaac Gym (2021), showed that you can train a gait in minutes with thousands of environments. Today it is legacy: the repositories checked by the course use Isaac Lab or mjlab.
06Practice
See the whole loop run before you try to make it learn.
E2 · More robots
10 minRun it again with --envs 32 --iters 10 --saida ~/fth/treino/ppo32 and compare the samples per iteration and the total time.
Done when: the first line says 32 ambientes × 24 passos = 768 amostras por iteração and ~/fth/treino/ppo32/modelo_10.pt exists.
E3 · Scale math
5 minWith the mjlab values (4,096 environments, 24 steps, 30,000 iterations), calculate how many samples a full training run collects. Write down how many times larger that is than your 20-iteration training.
Done when: you wrote 98,304 samples per iteration and about 2.9 billion in the full training (384 × 20 = 7,680 in yours: about 384 thousand times more).
07Lesson quiz
The explanation appears after your answer.
Question 1 of 3
What does the clip (clip 0.2) do in PPO?skill
Correct
The data was collected with the old policy; if the new one changes too much, the data stops being valid. The clip holds the step back, together with the KL-based rate adjustment.Not that one
The data was collected with the old policy; if the new one changes too much, the data stops being valid. The clip holds the step back, together with the KL-based rate adjustment.Review: One PPO iterationQuestion 2 of 3
The 20-iteration training ended with a negative reward. How do you know whether it "worked"?
Correct
On CPU and with 20 iterations, what you check is the loop. The reward value depends on the machine, the seed, and the duration; the format does not.Not that one
On CPU and with 20 iterations, what you check is the loop. The reward value depends on the machine, the seed, and the duration; the format does not.Review: The values, copied from mjlabQuestion 3 of 3
Why doesn't the course actually run mjlab or Isaac Lab?
Correct
The values and the names are the same; the scale changes. If you have a GPU, you run the shown commands in a separate environment, with the version pinned.Not that one
The values and the names are the same; the scale changes. If you have a GPU, you run the shown commands in a separate environment, with the version pinned.Review: Where real training runsQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Choose an option. The explanation appears after you try.
08In three sentences
LoopCollect 24 steps per robot, estimate the advantage with GAE, update with clipping and an adaptive rate.
Practice20 iterations on CPU write a checkpoint and a log; you check the format, not the value.
Scalemjlab uses the same values with 4,096 robots on the GPU and 30,000 iterations.