FTH 1 · Teaching it to walk|INEMA.CLUB-PRO

Lesson 3 · 16 min

Observation, action, reward, and end of episode

Before training, the problem has to become an environment: what the policy sees, what it sends, how much it earns, and when the episode ends. In this lesson you read the four parts in the "manager-based" format of Isaac Lab and mjlab, in a minimal version in plain MuJoCo, and you run one episode with three different policies.

Start the lesson
Open laptop on a dark workbench showing a blurred humanoid robot in 3D simulation, with a notebook and a mug beside it.

In this lesson you will

  1. 1"Manager-based" environment: named terms, not an opaque step()
  2. 2Observation of 96 numbers (480 with a history of 5) and action of 29
  3. 3End of episode: fall or time
  4. 4Random action falls in 1.3 s; the trained one makes it to 20 s

Goal: read the parts of a locomotion environment and run one episode with three different actions.

01The pieces of a locomotion environment

Click each block. In Isaac Lab and mjlab, each one is a list of named, weighted terms.

Click each block in the diagram to see what it does.

mjlab and Isaac Lab also have events (push, randomization) and a curriculum, which come in lessons 7 and 8.

02An episode with a random action

checar_ambiente.py prints the four parts and runs one episode. A random action is what a freshly created policy does, before any training.

Terminal · the four parts
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
python3 kit/checar_ambiente.py --acao aleatoria; echo "saída: $?"
Example from the repository
observação: 96 números = (base_ang_vel 3 + projected_gravity 3 + velocity_commands 3 + joint_pos_rel 29 + joint_vel_rel 29 + last_action 29) × histórico 1
ação: 29 números → alvo de cada junta = pose padrão + 0.25 × ação, a cada 20 ms
recompensa: track_linear_velocity (+2), track_angular_velocity (+2), upright (+1), pose (+1), dof_pos_limits (-1), action_rate_l2 (-0.1), foot_slip (-0.1)
terminações: queda (pelve inclinada > 70° ou abaixo de 0.3 m) · tempo (1000 passos = 20 s)
comando sorteado: vx = -0.48 m/s · vy = +0.19 m/s · giro = +0.17 rad/s
ação aleatoria: 67 passos · recompensa somada: -5.44
episódio terminou por: queda em t = 1.34 s
saída: 0

96 observation numbers, 29 action numbers, 7 reward terms, and two terminations. With a random action, it falls at 1.34 s with a negative reward: the penalties (limits, jitter) outweighed the rewards.

03Zero action and ready-made action

Same seed, same drawn command. Only what decides the action changes.

Terminal · zero × ready-made
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
for x in zero pronta; do python3 kit/checar_ambiente.py --acao $x | tail -2; done
Example from the repository
ação zero: 70 passos · recompensa somada: 3.78
episódio terminou por: queda em t = 1.40 s
ação pronta: 1000 passos · recompensa somada: 108.61
episódio terminou por: tempo em t = 20.00 s

Zero action also falls (1.40 s), but with a positive reward: it stayed a little upright and near the pose before falling. The ready-made policy reaches the time cutoff (1,000 steps) with a reward of 108.61. The episode that ends by tempo is what training wants.

04The same environment in mjlab (shown, not run here)

How the pieces appear in the mjlab velocity task configuration file (commit 8bb2bc9, 10/18/2026). It does not run here: it needs an NVIDIA GPU.

1. Classes

In velocity_env_cfg.py, observations, actions, commands, rewards, terminations, events, and curriculum are dictionaries of terms (ObservationTermCfg, RewardTermCfg, TerminationTermCfg…), each with a name, a function, and parameters.

05Practice

Know exactly what goes into and comes out of the policy.

E1 · The four parts

5 min

Run checar_ambiente.py with a random action and check the sum 3 + 3 + 3 + 29 + 29 + 29.

Done when: python3 kit/checar_ambiente.py --acao aleatoria; echo "saída: $?" → observação: 96 números and episódio terminou por: queda em t = 1.34 s.

E2 · Who makes it to the end

3 min

Run zero × ready-made and note why each episode ended.

Done when: the output shows episódio terminou por: queda em t = 1.40 s (zero) and episódio terminou por: tempo em t = 20.00 s (ready-made).

E3 · Another seed

5 min

Run with --seed 1, 2, and 3 (random action) and watch the command change. Write down why a fixed seed matters when comparing two trainings.

Done when: three different comando sorteado: lines, and the same seed repeated gives the same output.

06Lesson quiz

The explanation appears after your answer.

Question 1 of 3

Why does the ready-made policy receive 480 numbers and not 96?skill

07In three sentences

PiecesCommand, observation, action, reward and termination, each one as a list of named terms.
Sizes96 observation numbers (480 with history) and 29 action numbers, target = default pose + 0.25 × action.
MeasureRandom falls at 1.34 s; the trained one reaches 20 s with 108.61.

Going further

mjlab: the velocity task (velocity_env_cfg.py)mjlab (arXiv 2601.22074)unitree_rl_lab: the G1 29-DoF velocity environmentFTH Course 0: sensors (IMU, gravity in the body frame)
Previous lessonNext lesson