LeRobot: Hugging Face’s End-to-End Framework

Detailed content

🔹 What LeRobot is: the end-to-end open philosophy

LeRobot is Hugging Face’s effort to do for robotics what the transformers did it with NLP: democratize. In a single PyTorch repo, you get standardized datasets, reproducible SOTA policies, sim environments, and low-cost hardware drivers. The goal is to eliminate the fragmentation that held the field back.

📊 The four pillars

Datasets — a single format (LeRobotDataset) versioned on the Hub.
Policies — ACT, Diffusion Policy, π0, VQ-BeT with public checkpoints.
Yes + envs — gym wrappers (ALOHA, PushT, Xarm) for evaluation.
Hardware — SO-100/SO-101, Koch, ALOHA with teleop and recording scripts.

⚡ Practical tip

Everything revolves around the CLI/entry points (lerobot-record, lerobot-train, lerobot-eval). Learn the three commands and you can already run the full cycle on a desktop arm.

Open

Public code + weights.

End-to-end

Data → policy → robot.

Hub

Version the data and ckpt.

Accessible

Low-cost hardware.

🔹 LeRobotDataset: parquet + video format

O LeRobotDataset separates lightweight from heavyweight data: state, actions, and timestamps go in parquet (columnar, fast), while images live in mp4 encoded and decoded on demand. Each episode is a trajectory; features describe the shape and dtype of each key.

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

ds = LeRobotDataset("lerobot/svla_so101_pickplace")
print(ds.features.keys())
# observation.images.top, observation.state, action, timestamp, ...

sample = ds[0]
sample["observation.images.top"].shape   # (3, H, W) tensor já decodificado
sample["action"].shape                    # (action_dim,)

# delta_timestamps -> janelas temporais (chunking) automáticas
ds = LeRobotDataset(repo_id, delta_timestamps={"action": [t/30 for t in range(16)]})

📊 Anatomy of an episode

observation.images.* — one key per camera (top, wrist), in mp4.
observation.state — joint positions / gripper pose.
action — target command for the next step.
episode_index / frame_index — trajectory boundaries.

parquet

Columnar state/action.

mp4

Lazy image decoding.

features

Schema shape+dtype.

delta_timestamps

Time windows.

🔹 Ready-to-use policies: ACT, Diffusion, π0, VQ-BeT

LeRobot packages policies from different classes behind the same interface. ACT (Action Chunking Transformer, from ALOHA) is the workhorse for manipulation with few demos; Diffusion Policy capture multimodality; π0 brings generalist flow matching; VQ-BeT uses quantized vector tokens.

# treinar ACT no dataset do SO-101
lerobot-train \
  --policy.type=act \
  --dataset.repo_id=lerobot/svla_so101_pickplace \
  --batch_size=8 --steps=100000 \
  --output_dir=outputs/act_so101 \
  --policy.device=cuda --wandb.enable=true

# trocar para diffusion: --policy.type=diffusion
# fine-tune do π0: --policy.type=pi0 --policy.pretrained_path=lerobot/pi0

✓ Do

✓Start with ACT: few hyperparameters, fast training, great baseline.
✓Use pretrained π0 when you need semantic generalization.
✓Match the policy chunk size to the robot's frequency.

✗ Avoid

✗Diffusion Policy with 10 demos and wait for convergence.
✗Switch policies without rechecking action normalization.
✗Train π0 from scratch without a powerful GPU — start from the checkpoint.

ACT

Transformer + VAE chunking.

Diffusion

Trajectory denoising.

π0

General-purpose flow matching.

VQ-BeT

Quantized tokens.

🔹 Supported hardware: SO-101, Koch, ALOHA

O SO-101 (successor to the SO-100, a project by The Robot Studio + Hugging Face) is the reference arm: ~6 DOF, Feetech servos, costing around $100–150. Data collection uses a leader-follower — you move a "leader" arm and the "follower" replicates it, recording state+action synchronized with the camera.

Assemble and calibrate

Identify motor ports, run lerobot-calibrate to map each joint’s range.

Teleoperate

The leader moves; the follower copies. Cameras (top + wrist) capture the scene in real time.

Record episodes

30–50 demos are usually enough for ACT on a simple pick-and-place task.

⚡ Practical tip

Consistency > quantity. Clean demos, consistent lighting, and deliberate variation in the object's position teach generalization better than 200 messy episodes.

SO-101

~$100 arm, 6 DOF.

Leader-follower

Copy-based teleoperation.

Calibration

Map joint ranges.

ALOHA

Reference bimanual setup.

🔹 Workflow: record → train → evaluate → deploy

The full cycle takes three commands and an inference loop. You record demos, train a policy, evaluate it on the robot (or in simulation), and deploy running the policy in a closed loop at ~30 Hz.

# 1) gravar 50 demos no SO-101
lerobot-record \
  --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
  --teleop.type=so101_leader --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=eu/so101_pegar_cubo --dataset.num_episodes=50

# 2) treinar
lerobot-train --policy.type=act --dataset.repo_id=eu/so101_pegar_cubo --steps=80000

# 3) avaliar/deploy: rodar a policy no follower
lerobot-record --policy.path=outputs/act/checkpoints/last \
  --robot.type=so101_follower --dataset.num_episodes=10  # eval rollouts

📊 Training health signals

loss decreasing but the success rate stays flat → a normalization/camera problem.
overfit with few demos → increase diversity, not just steps.
success in sim, real-world failure → sim-to-real gap (Track 3).

⚡ Practical tip

Always evaluate with the robot in the same configuration from the recording (lighting, camera positions, background). Changing the scene between training and deployment is the #1 cause of a policy that "worked yesterday" failing today — and it's not a bug, it's distribution shift.

record

Demo collection.

train

Trains the policy.

eval rollout

Measures success rate.

deploy

30 Hz closed loop.

✓ Do

✓Evaluate with rollouts on the robot, not just the training loss.
✓Log rollout videos to diagnose failures.
✓Version the dataset + config + checkpoint together.

✗ Avoid

✗Increasing steps when the problem is a lack of diversity.
✗Relying only on falling loss as a sign of success.
✗Change the scene between recording and deployment.

🔹 Community and reproducibility

LeRobot’s value scales with the community: thousands of public datasets and checkpoints on Hub, each versioned and with a descriptive card. You can reproduce a paper by downloading the exact dataset and config, and contribute by publishing your SO-101 demos for others to train on.

# publicar seu dataset no Hub
huggingface-cli login
lerobot-record ... --dataset.push_to_hub=true --dataset.repo_id=eu/minha_tarefa

# visualizar qualquer dataset no navegador
# https://huggingface.co/spaces/lerobot/visualize_dataset

⚡ Practical tip

Before recording from scratch, search the Hub: there may be an SO-101 dataset for your task. Co-training with public data + your demos improves generalization — the perfect segue into the datasets module (2.6).

Hub

Datasets + checkpoints.

Reproducible

Same data + config.

push_to_hub

Contribute demos.

visualize

Browser inspection.

✅ Module summary

✓

LeRobot unifies — data, policies, simulation, and hardware in one open repo.

✓

LeRobotDataset — parquet for state/action, mp4 for images, typed features.

✓

ACT as a baseline — then Diffusion/π0/VQ-BeT depending on the task.

✓

3-command cycle — record, train, eval/deploy on the SO-101.

✓

Hub = reproducibility — reuse and contribute datasets.

Next module

2.3 — Simulators: MuJoCo/MJX, Isaac Sim/Lab, and Habitat, where we generate cheap, safe data at scale.

← Back to the track Next module →