E1 · Evaluate 10 episodes
15 minInstall lerobot[pusht] and run the lerobot-eval from the box (cpu or cuda).
Done when: the terminal shows pc_success and n_episodes: 10, and outputs/eval/act_200/eval_info.json exists.
Lesson 6 · 16 min
Simulate before you break things: sim, evaluation, and sim-to-real
Testing on the real robot costs time, parts, and patience. The simulator lets you fail a thousand times for free and gives you one number: the success rate. But the number only counts if the test is hard enough and if the simulated world looks like the real one.
Start the lesson
In this lesson you will
- 1The 2026 simulators: MuJoCo, Isaac Sim/Lab, and Newton
- 2Run lerobot-eval and read the success rate
- 3Why a saturated benchmark doesn't separate models
- 4Randomization and system identification against the reality gap
Goal: evaluate a policy in simulation and know why LIBERO alone is no longer enough.
01From simulator to the real world
Click each step.
Reality gap = the distance between what the policy sees and feels in the simulator and on the real robot.
Simulation to fail cheaply; benchmark to compare; real world to decide.
02Three ideas so you don't fool yourself
Click each one.
1. Saturated benchmark
In 2026 several models score above 97% on LIBERO: OpenVLA-OFT 97.1%, X-VLA 98.1%, MolmoAct 2-Think 98.1%. When everyone gets a top score, the test doesn't separate anyone.
That's why harder tests (LIBERO-plus, RoboTwin 2.0) and real-world evaluation (RoboArena, MolmoBot) are showing up.
2. Randomization
Instead of a perfect simulator, you deliberately vary color, lighting, friction, mass, and camera position in every episode.
A policy that learns with that much variation treats the real world as just one more variation.
3. Identification
You measure the real robot (mass, friction, motor delay) and tune the simulator to match.
MuJoCo 3.5.0 (Feb/2026) brought a Python toolbox for this, plus actuator and sensor delays.
In practice
Use both together: measure what you can measure and randomize the rest.
03Evaluate your policy on PushT
Install the simulated environment and run the policy you trained in lesson 5.
Terminal · install the PushT environment
pip install "lerobot[pusht]==0.6.1"
Terminal · evaluate on 10 episodes
lerobot-eval \ --policy.path=outputs/train/act_pusht_teste/checkpoints/last/pretrained_model \ --env.type=pusht \ --eval.n_episodes=10 \ --eval.batch_size=10 \ --eval.use_async_envs=false \ --policy.device=cuda \ --output_dir=outputs/eval/act_200
Terminal · see the result and the videos
python -c "import json; d = json.load(open('outputs/eval/act_200/eval_info.json')); print(d['overall'])"
ls outputs/eval/act_200/videos/pusht_0/Example from the repository
INFO ... Overall Aggregated Metrics:
INFO ... {'avg_sum_reward': 11.107..., 'avg_max_reward': 0.0824..., 'pc_success': 0.0, 'n_episodes': 10, 'eval_s': 9.46...}
INFO ... End of eval
# same evaluation with the 5000-step checkpoint (E3 from lesson 5), 20 episodes:
{'avg_sum_reward': 19.45..., 'avg_max_reward': 0.275..., 'pc_success': 0.0, 'n_episodes': 20, 'eval_s': 14.0...}LeRobot 0.6.1. With 200 steps: 13 s total, on both GPU and CPU only (the evaluation is light). pc_success is the success rate in %; avg_max_reward shows how close the policy got to the target (higher is better). Both short trainings give 0% success: the policy still doesn't push the T to the target, but with 5000 steps it gets closer (0.08 → 0.28).
- "Namespace gym_pusht not found": happened here with more than one environment in parallel. The
--eval.use_async_envs=falseoption fixed it. - Old checkpoint from the Hub:
lerobot/diffusion_pushtneeds a format migration in 0.6 and doesn't run directly. Evaluate your own. - Reading 0% as "it doesn't work": with few steps it's expected. Look at the reward and the videos to see if it improved.
- No display: on a server without a monitor, use
SDL_VIDEODRIVER=dummybefore the command.
04Practice: your first success rate
Use the checkpoint from lesson 5.
E2 · Watch the policy
10 minOpen two videos in outputs/eval/act_200/videos/pusht_0/ and describe in one sentence what the policy does wrong.
Done when: your sentence says where it fails (doesn't reach the block, pushes it the wrong way, stops midway).
E3 · Compare two checkpoints
10 minIf you did the 5000-step training, evaluate it too (change --policy.path and --output_dir) and compare avg_max_reward.
Done when: you have both numbers side by side and can say which policy got closer to the target.
05Test what you learned
The explanation appears after your answer.
Question 1 of 4
What does pc_success in lerobot-eval measure?skill
Correct
It's the success rate over N episodes. Loss comes from training; success comes from the task. They are different numbers.Not this one
It's the success rate over N episodes. Loss comes from training; success comes from the task. They are different numbers.Review: Evaluate your policy on PushTQuestion 2 of 4
Why is LIBERO alone no longer enough to compare VLAs?
Correct
With everyone near 100%, the test doesn't separate them. Use harder versions (LIBERO-plus, RoboTwin 2.0) or real-world evaluation (RoboArena).Not this one
With everyone near 100%, the test doesn't separate them. Use harder versions (LIBERO-plus, RoboTwin 2.0) or real-world evaluation (RoboArena).Review: Three ideas so you don't fool yourselfQuestion 3 of 4
Deliberately varying lighting, friction, and mass in the simulator on every episode is…
Correct
Randomization teaches the policy to tolerate variation. System identification does the opposite: it measures the robot and tunes the simulator to match it.Not this one
Randomization teaches the policy to tolerate variation. System identification does the opposite: it measures the robot and tunes the simulator to match it.Review: Three ideas so you don't fool yourselfQuestion 4 of 4
Your policy had 0% success and avg_max_reward went from 0.08 to 0.28 with more training. What do you conclude?
Correct
Success is binary per episode; the reward shows how close it got. Short training is for learning the loop, not for solving the task.Not this one
Success is binary per episode; the reward shows how close it got. Short training is for learning the loop, not for solving the task.Review: Evaluate your policy on PushTQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Pick an option. The explanation appears after your attempt.
06Summary
Simulate, compare, decideSimulator to fail cheaply, benchmark to compare, real robot to decide.
One commandlerobot-eval runs N episodes and returns pc_success and videos.
Reality gapRandomize what you can't measure and measure what you can (sysid).