MODULE 2.2

🤗 LeRobot: End-to-End Pipeline

HuggingFace’s open-source framework that democratizes robotics: data collection, training, and deployment in a single tool. With 12k stars on GitHub and 58k+ datasets, LeRobot is the most accessible entry point for practical VLA.

Topics

~50

Minutes

Technical

Level

Practical

Type

Illustrative diagram — end-to-end LeRobot flow: collection → dataset → training → evaluation → deploy.

🤗 LeRobot: 12k stars, 58k+ datasets

LeRobot is the HuggingFace open-source framework for end-to-end robotics. Released in 2024, it unifies data collection, policy training, and deployment on real robots in a single tool with extensive documentation and an active community.

🏗️ What LeRobot solves

Before LeRobot, each lab had its own ad hoc pipeline: proprietary data collection scripts, incompatible dataset formats, isolated training pipelines. LeRobot standardizes everything with a cohesive API — from hardware to the trained model — and publishes it all to the HuggingFace Hub for reuse.

✓ What LeRobot includes

✓Data collection scripts with teleoperation (lerobot/record)
✓Pre-implemented policies: ACT, Diffusion Policy, TDMPC, VQ-BeT
✓Native integration with HuggingFace Hub (58k+ datasets)
✓Support for multiple robots: SO-100, Koch v1.1, Aloha
✓Episode viewer and evaluation metrics

✗ What LeRobot doesn’t do

✗Is not a simulator — you need MuJoCo/Isaac Sim for sim-to-real
✗Does not replace ROS2 for low-level control in production
✗Does not train vision models from scratch (uses pretrained backbones)
✗Does not guarantee safety in uncontrolled environments

⚡ Quick Installation

pip install lerobot

# Verificar instalação
python -c "import lerobot; print(lerobot.__version__)"

# Listar políticas disponíveis
python -c "from lerobot.common.policies import *; print('ACT, Diffusion, TDMPC, VQ-BeT')"

Key concepts

LeRobotDataset

Main class that encapsulates a dataset in RLDS format with synchronized video, actions, and metadata.

lerobot/scripts/train.py

Unified training script that loads a YAML configuration and automatically saves checkpoints.

Hub integration

Push and pull datasets and models directly from the HuggingFace Hub with huggingface-cli login.

Multi-robot support

Ready-to-use configurations for SO-100, Koch v1.1, and Aloha; add a new robot via calibration YAML.

🎮 Data Collection via Teleoperation

Teleoperation is the technique of a human remotely controlling the robot to generate quality demonstrations. LeRobot synchronously records video from the cameras and actuator positions, producing episodes with aligned timestamps that feed supervised training.

💡 The 50–100 episode rule

For simple tasks (pick-and-place, stacking blocks), 50-100 high-quality episodes outperform 500 poor-quality episodes. Consistency and coverage of variations (angles, object positions) matter more than raw volume.

🖥️ Recording episodes with LeRobot

# Calibração do robô (uma vez)
python lerobot/scripts/control_robot.py calibrate \
  --robot-path lerobot/configs/robot/so100.yaml

# Gravar dataset de demonstrações
python lerobot/scripts/control_robot.py record \
  --robot-path lerobot/configs/robot/so100.yaml \
  --fps 30 \
  --repo-id seu-usuario/pick-place-dataset \
  --num-episodes 50 \
  --warmup-time-s 5

# Replay para validar qualidade
python lerobot/scripts/control_robot.py replay \
  --robot-path lerobot/configs/robot/so100.yaml \
  --repo-id seu-usuario/pick-place-dataset \
  --episode 0

✓ Ideal leader-follower setup

✓2 cameras: one global, one on the robot's wrist
✓30 Hz synchronized capture of cameras + actuators
✓Intrinsic/extrinsic calibration before each session
✓Check each episode's replay before proceeding

✗ Common data collection errors

✗Abrupt movements or hesitation in the demonstration
✗Cameras with poor focus or inconsistent lighting
✗Objects always in the same position (little variation)
✗Not filtering failed episodes before training

Key concepts

Leader-follower

The “leader” robot (controlled by a human) commands the “follower” (which performs the task). Both have sensors to capture the real state.

Episode recording

Each demonstration is an episode: a sequence of observations (images + positions) and actions with aligned timestamps.

Data quality filtering

Remove episodes where the robot failed, made anomalous movements, or the camera was obstructed.

🧠 Policies: ACT, Diffusion Policy, TDMPC

LeRobot implements multiple policy architectures, each with different trade-offs between inference speed, motion quality, and training complexity. Choosing the right policy for your task is as important as data quality.

Evolution of policy architectures

ACT — Action Chunking with Transformers (2023)

Predicts k actions at once (action chunk) instead of one at a time. Reduces compounding errors. Fast inference (~10ms), ideal for real-time control at 50Hz.

chunk_size=100 Transformer encoder-decoder

Diffusion Policy (2023)

Models action distributions with DDPM. Captures multimodality (multiple ways to solve the task). Slower at inference (10-100 denoising steps), but produces smoother movements.

num_inference_steps=10 Conditional UNet

TDMPC — TD-MPC2 (2024)

World model + planning: learns a model of the environment’s dynamics and uses MPC (Model Predictive Control) to plan actions. Better generalization, but requires more data and compute.

horizon=5 Latent world model

💡 Which policy should you choose?

Start with ACT for simple manipulation tasks — it is the fastest for iteration. Use Diffusion Policy when you need multimodal movements (e.g., a task that can be solved in multiple ways). Reserve TDMPC for long-horizon tasks with sim-to-real.

Key concepts

Action chunking

Predict multiple future actions at once to reduce compounding errors during sequential execution.

DDPM (Denoising Diffusion)

Progressive denoising process that generates high-quality samples from complex action distributions.

VQ-BeT

Discrete vector tokenization of actions — treats actions as tokens, enabling the use of LLM architectures.

Multimodality

Ability to learn multiple ways to solve a task instead of the “average” of all demonstrations.

⚙️ GPU Training: YAML Config and Checkpoints

LeRobot uses YAML files with Hydra for hierarchical training configuration. Each run automatically saves checkpoints and can be resumed, enabling rapid iteration without rework.

🚀 Starting a training run

# Treinar ACT no seu dataset
python lerobot/scripts/train.py \
  policy=act \
  env=aloha \
  dataset_repo_id=seu-usuario/pick-place-dataset \
  training.num_steps=80000 \
  training.batch_size=8 \
  wandb.enable=true \
  hydra.run.dir=outputs/train/act_pick_place

# Retomar do checkpoint
python lerobot/scripts/train.py \
  resume=true \
  resume_path=outputs/train/act_pick_place/checkpoints/last

# Multi-GPU com DDP (2 GPUs)
torchrun --nproc_per_node=2 lerobot/scripts/train.py \
  policy=diffusion \
  dataset_repo_id=seu-usuario/pick-place-dataset

Illustrative diagram — training flow: YAML → GPU → checkpoint → W&B metrics.

Key concepts

Gradient accumulation

Accumulate gradients over N batches before the update — allows a larger effective batch size than VRAM permits.

Mixed precision (BF16)

Use BFloat16 for forward/backward passes and FP32 for weight updates — 2x faster with similar memory usage.

DDP (DistributedDataParallel)

PyTorch multi-GPU strategy: each GPU processes a mini-batch, and gradients are synchronized via AllReduce.

resume_from_checkpoint

Resume training from a saved checkpoint, preserving optimizer and learning rate scheduler state.

📊 Weights & Biases: Experiment Tracking

W&B is the standard platform for tracking ML experiments. LeRobot has native integration — just add wandb.enable=true in the config YAML to centralize metrics, loss curves, and evaluation videos.

💡 No tracking = black box

Training VLAs without structured tracking is like walking blind. You need to know: which run had the best success rate? Which learning rate diverged? At what step did the model stop improving? W&B answers all of this with side-by-side comparison charts.

📈 Configuring W&B in LeRobot

# Autenticar no W&B
wandb login

# No YAML de config (config/default.yaml)
wandb:
  enable: true
  project: "lerobot-experiments"
  entity: "seu-usuario"
  name: "act-pick-place-v1"

# Ou via override na linha de comando
python lerobot/scripts/train.py \
  policy=act \
  wandb.enable=true \
  wandb.project=lerobot-experiments \
  wandb.name=act_run_001

# Sweeps para hyperparameter search
wandb sweep sweep_config.yaml
wandb agent usuario/projeto/sweep_id

✓ Essential metrics to monitor

✓train/loss — should decrease monotonically (with noise)
✓eval/success_rate — percentage of successful episodes
✓eval/video — recorded videos of the robot being evaluated
✓Learning rate schedule — confirm that it decays as expected

✗ Dashboard warning signs

✗Loss rises abruptly — possibly a high learning rate
✗Success rate plateaued after 20k steps — insufficient dataset
✗Gradient norm exploding — add gradient clipping
✗GPU utilization below 70% — bottleneck in data loading

Key concepts

wandb.Sweep

Automatic hyperparameter search: define a search space (lr, batch_size) and W&B schedules and tracks all runs.

Model Registry

Version model checkpoints with semantic tags (best, staging, production) for traceability.

Tables for trajectories

Log action and observation sequences as W&B tables for qualitative behavior analysis.

wandb.config

Automatically record all run hyperparameters for reproducibility and experiment comparison.

🐳 Docker and Reproducibility in Robotics

Reproducibility is one of the biggest challenges in robotics research. CUDA driver versions, Python dependencies, and hardware configurations create fragile environments. Docker containerizes everything — a Dockerfile ensures anyone (or any CI server) can reproduce your training exactly.

💡 NVIDIA NGC: ready-to-use containers

NVIDIA maintains the NGC (NVIDIA GPU Cloud) with optimized PyTorch+CUDA containers for various GPUs. LeRobot provides Dockerfiles that extend these containers—you don’t need to configure CUDA manually.

🐳 Multi-stage Dockerfile for LeRobot

# Etapa 1: base com CUDA
FROM nvcr.io/nvidia/pytorch:24.01-py3 AS base
WORKDIR /workspace

# Etapa 2: dependências
FROM base AS deps
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install lerobot

# Etapa 3: código
FROM deps AS runtime
COPY . .
ENV HF_HOME=/workspace/.cache/huggingface

# Rodar treinamento com GPU passthrough
# docker run --gpus all -v /data/datasets:/workspace/datasets \
#   -v /data/checkpoints:/workspace/outputs \
#   lerobot-train:latest \
#   python lerobot/scripts/train.py policy=act ...

✓ Docker best practices for robotics

✓Multi-stage Dockerfile to separate build from runtime
✓Build datasets as volumes (-v) — don't copy into the image
✓Pin the base container version (:24.01 no :latest)
✓Use devcontainers in VS Code for development

✗ Common pitfalls

✗Copying datasets into the Docker image (wasted gigabytes)
✗Forget --gpus all in docker run (trains on CPU)
✗Use :latest — image changes and breaks reproducibility
✗Does not version the Dockerfile alongside the experiment code

Key concepts

nvidia-docker2

Plugin that lets Docker containers access NVIDIA GPUs via --gpus all without manual CUDA configuration.

NVIDIA NGC

Registry of GPU-optimized containers: PyTorch, TensorFlow, TensorRT — a reliable foundation for reproducible experiments.

devcontainers

VS Code integration that opens the editor inside the Docker container—the same development and production environment.

✅ Module Summary

✓

LeRobot = data collection + training + deployment — HuggingFace’s unified framework with 12k stars and 58k+ public datasets on the Hub.

✓

Teleoperation produces high-quality data — 50-100 consistent episodes outperform 500 poor ones; calibration and camera setup are critical.

✓

Policies with distinct trade-offs — ACT for speed, Diffusion Policy for multimodal smoothness, TDMPC for long horizons.

✓

W&B + Docker = reproducibility — experiment tracking and containerization ensure that anyone can reproduce results.

Next module

2.3 — Simulators: MuJoCo, Isaac Sim, PyBullet

← Back to the Learning Path Next module →