E1 · Test with a push
5 minWrite your pass criterion (e.g., 18/20) and only then run the evaluation with a push of 1.0 m/s.
Done when: python3 kit/avaliar.py --episodios 20 --empurrao 1.0; echo "saída: $?" → sobreviveu: 20/20 and saída: 0.
Lesson 7 · 16 min
Evaluating for real: speed, turning and push
A policy is only ready when it passes a test you wrote before looking at the result. In this lesson you evaluate the ready-made policy over 20 episodes with fixed seeds: speed error, turning and pushes of 1.0, 1.5 and 2.0 m/s. Then you run your checkpoint from lesson 5 through the same test.
Start the lesson
In this lesson you will
- 1Speed error: 0.05 m/s between requested and measured
- 2Push of 1.0 m/s: 20/20 standing; 1.5: 16/20; 2.0: 9/20
- 3Fixed seeds: the same 20 episodes for any policy
- 4Where evaluation in sim ends and sim2sim begins (course 2)
Goal: evaluate a policy over 20 episodes with fixed seeds, measuring speed error and survival to pushes of increasing strength.
01What a walking test measures
Click each block. avaliar.py runs all four in every episode.
Click each block in the diagram to see what it does.
The test is written before the result: number of episodes, command range, strength, and pass criterion (e.g., 18/20).
02The ready-made policy in the test
20 episodes of 10 s, push of 1.0 m/s. Exits 0 when all survive.
Terminal · evaluate with a push
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
python3 kit/avaliar.py --episodios 20 --empurrao 1.0; echo "saída: $?"Terminal · stronger pushes
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
for e in 1.5 2.0; do python3 kit/avaliar.py --episodios 20 --empurrao $e | tail -1; doneExample from the repository
política: unitree_rl_lab velocity v0 (ONNX) · 20 episódios de 10 s · giro +0.0 rad/s · empurrão de 1.0 m/s aos 4 s erro médio de velocidade (quem não caiu): 0.05 m/s sobreviveu: 20/20 saída: 0
20 out of 20 stayed up with a 1.0 m/s push, above the training (up to 0.5 m/s per axis, about 0.71 m/s on the diagonal), with an average velocity error of 0.05 m/s.
sobreviveu: 16/20 sobreviveu: 9/20
At 1.5 m/s, 16 out of 20; at 2.0 m/s, 9 out of 20. The margin runs out between 1.0 and 1.5 m/s. This kind of curve, not a single number, tells you how much the policy can take.
03Your checkpoint in the same test
avaliar.py accepts a .pt trained by the kit in place of the ready-made policy. Same seeds.
Terminal · your policy
F=${FTH:-$HOME/fth}; cd $F/locomocao && . $F/.venv/bin/activate
python3 kit/avaliar.py --politica $F/treino/ppo/modelo_20.pt --episodios 5 | tail -2Example from the repository
quedas: ep 0 em 1.3 s, ep 1 em 1.4 s, ep 2 em 1.4 s, ep 3 em 1.4 s, ep 4 em 1.3 s sobreviveu: 0/5
0 out of 5, all falls before 1.5 s, even without a push. That is expected for 20 iterations: the test confirms what the curves already said. The value matters less than the habit: every policy, ready-made or yours, goes through the same test.
04How far simulation evaluation goes
What this test proves and what is left for course 2.
1. Proves
That the policy tracks the command and withstands pushes in MuJoCo, with the Menagerie G1. The ready-made policy was trained in Isaac Lab: passing here is already a first sign of robustness.
2. Does not prove
That it walks on the robot. Different delay, actuator, friction, and mass are the subject of course 2, which measures this gap and runs the policy through a sim2sim gate with DDS before it touches the floor.
3. Concept
The concept of evaluating in simulation and of the reality gap, for manipulation policies, is in vla-v72, lesson 6.
05Practice
Write the criterion before seeing the number.
E2 · The margin
5 minRun 1.5 and 2.0 m/s and write down between which strengths the policy's margin runs out.
Done when: the two lines show sobreviveu: 16/20 and sobreviveu: 9/20; you wrote "between 1.0 and 1.5 m/s".
E3 · Walking in a curve
5 minRun the same test with --giro 0.2 and compare the velocity error with the straight-line one.
Done when: python3 kit/avaliar.py --episodios 20 --giro 0.2 --empurrao 1.0 → the first line shows giro +0.2 rad/s and the last one sobreviveu: 20/20 (on 10/19/2026, error of 0.04 m/s).
06Lesson quiz
The explanation appears after your answer.
Question 1 of 3
Why are the test seeds fixed?skill
Correct
With different seeds, a difference from 16/20 to 18/20 could be just luck. Fixing them makes the comparison fair and repeatable.Not that one
With different seeds, a difference from 16/20 to 18/20 could be just luck. Fixing them makes the comparison fair and repeatable.Review: What a walking test measuresQuestion 2 of 3
20/20 at 1.0 m/s, 16/20 at 1.5, and 9/20 at 2.0. What does this show?
Correct
Training used pushes of up to 0.5 m/s per axis (about 0.71 m/s on the diagonal); withstanding 1.0 m/s is margin. The curve of increasing force tells you how far it goes.Not that one
Training used pushes of up to 0.5 m/s per axis (about 0.71 m/s on the diagonal); withstanding 1.0 m/s is margin. The curve of increasing force tells you how far it goes.Review: The ready-made policy in the testQuestion 3 of 3
The policy passed 20/20 in MuJoCo. What is still missing before the robot?
Correct
Evaluating in sim proves the behavior in the simulator. The path to the ground, with the gap measured and a ladder of tests, is course 2.Not that one
Evaluating in sim proves the behavior in the simulator. The path to the ground, with the gap measured and a ladder of tests, is course 2.Review: How far simulation evaluation goesQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Choose an option. The explanation appears after you try.
07In three sentences
TestCriterion written first, 20 episodes with fixed seeds, tracking and push.
MeasureThe ready-made policy: 20/20 at 1.0 m/s, 16/20 at 1.5, 9/20 at 2.0.
LimitPassing in sim is the first step; sim2sim and the gap are in course 2.