🎓 Learning by demonstration

In module 5.3, you made the follower copy the leader in real time. Now the idea is to take it one step further: record these demonstrations and train a neural network that learns the connection “what I see and where my joints are → what the next movement should be.” Once trained, it controls the follower without anyone at the leader. This is called imitation learning.

How to read this map: the green arrows show one cycle; the glowing block (training) is the only part that needs a GPU. The cyan arrow marks the difference between those who succeed and those who give up: evaluation shows where the policy fails, and the fix is to record more examples of exactly that situation.

Why learn this

Programming the arm to “pick and place” works until the cube moves 3 cm from its spot. The learned policy looks at the camera at every moment and adjusts its path. That’s how labs train arms today, and the SO-101 with LeRobot brings this method to your desk.

Key concepts

Each recorded demonstration is an episode. The command lerobot-record connects the leader, follower, and cameras, and saves three things at each frame: the images, the follower’s current joint positions (the state) and the position the leader commanded (the action). Between episodes, there’s a reset period so you can set up the table.

How to read this map: each column is a frame. The top row shows what the camera saw (the cube gets closer, is picked up, and appears in the box). The rows below show the numbers the network will learn to connect: given the state and image, predict the action. The text instruction applies to the entire episode.

Step-by-step: your first recording

Goal: record episodes for the task “put the cube in the box” with a front-facing camera. Replace the ports, the id and your Hugging Face username (without the < > symbols).

# Record demonstrations with the SO-101 pair (Linux; on Windows the port is COM3, COM4...)
HF_USER=<seu_usuario_hf>          # replace with your Hugging Face username
DATASET=cubo-na-caixa-teste        # dataset name

lerobot-record \
  --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=seguidor \
  --robot.cameras="{ frente: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --teleop.type=so101_leader --teleop.port=/dev/ttyACM1 --teleop.id=lider \
  --dataset.repo_id=${HF_USER}/${DATASET} \
  --dataset.single_task="Put the red cube in the gray box." \
  --dataset.num_episodes=2 \
  --dataset.episode_time_s=20 \
  --dataset.reset_time_s=10 \
  --dataset.push_to_hub=false \
  --display_data=true

Why learn this

The policy can never be better than the demonstrations. If you record calmly and consistently, you train once; if you record in a hurry, you train five times and still don’t understand why it doesn’t work.

Key concepts

LeRobot saves each dataset in a standard format: tables with state and action, camera videos, and a metadata file. This same format can be uploaded to the Hugging Face Hub, where you can train on a machine with a GPU (or in Colab) without copying files by hand, and where other people can use your data if you want.

How to read this map: the name at the top is the repo_id. The tables (middle) are lightweight; the videos (right, in cyan) take up almost all the space. If the dataset is too large, reduce the resolution or number of cameras, not the number of episodes.

Goal: log in, check a recorded episode, and upload the dataset to the Hub as private. Run this in the same Python environment where you installed LeRobot.

hf auth login                       # paste the write token (once per machine)
lerobot-dataset-viz --repo-id <seu_usuario_hf>/<dataset> --episode-index 0

# Open the local dataset, show its size, and upload it to the Hub as private.
import sys
from lerobot.datasets.lerobot_dataset import LeRobotDataset

repo_id = sys.argv[1]                    # e.g.: <your_hf_username>/<dataset>
ds = LeRobotDataset(repo_id)             # reads from ~/.cache/huggingface/lerobot/
print("episodes:", ds.num_episodes, "| quadros:", ds.num_frames, "| fps:", ds.fps)
print("colunas:", list(ds.features))
ds.push_to_hub(private=True)             # private: only you can see it
print("uploaded: https://huggingface.co/datasets/" + repo_id)

Why learn this

Separating "where you record" from "where you train" solves the problem for people without a GPU at home: record on the laptop next to the arm, then train in Colab or in the lab. A well-documented dataset also becomes lesson material for the whole class.

Key concepts

A ACT (Action Chunking with Transformers) is the recommended policy to start with: it is small, trains quickly, and works well with 50 episodes of one task. The trick is in the name: instead of predicting one movement at a time, it predicts a chunk of several consecutive movements, which makes the arm smoother and less hesitant.

How to read this map: on the left is everything the network sees at a given moment. On the right, the bars represent future movements: the solid ones are executed, while the outlined ones are discarded because by then the camera will have new information. The cyan arrow closes the loop, as in the sense-think-act cycle in 1.1.

Goal: train an ACT on your dataset on a machine with an NVIDIA GPU. On a Mac with an Apple chip, replace cuda with mps; without a GPU, use Colab (there’s a ready-to-use notebook in the LeRobot documentation).

# Train the ACT policy (check the flags with: lerobot-train --help)
HF_USER=<seu_usuario_hf>
DATASET=<dataset>

lerobot-train \
  --dataset.repo_id=${HF_USER}/${DATASET} \
  --policy.type=act \
  --policy.device=cuda \
  --output_dir=outputs/train/act-${DATASET} \
  --job_name=act-${DATASET} \
  --batch_size=8 \
  --steps=20000 \
  --wandb.enable=false \
  --policy.push_to_hub=false

Why learn this

Knowing how to read training results helps you avoid wasting GPU time: you learn when to stop after the loss levels off, test intermediate checkpoints, and tell the difference between "not enough data" and "not enough training."

Key concepts

Evaluation means letting the policy control the follower, without the leader, and counting how many times it succeeds. In the current version of LeRobot, you do this with lerobot-rollout; earlier versions used lerobot-record itself with the specified policy. Check which one applies to your installation with --help.

Goal: run the trained policy on the follower for 30 seconds, using the same camera and the same instruction as in the recording.

# Run the policy on the real arm (check the flags with: lerobot-rollout --help)
POLITICA=outputs/train/act-<dataset>/checkpoints/last/pretrained_model

lerobot-rollout \
  --strategy.type=base \
  --policy.path=${POLITICA} \
  --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=seguidor \
  --robot.cameras="{ frente: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --task="Put the red cube in the gray box." \
  --duration=30

Evaluation spreadsheet (20 attempts)

When a failure pattern appears (for example, "it always misses when the cube is on the left"), record 10 to 20 new episodes in that area and train again. LeRobot also has a mode where you take control with the leader when the policy makes a mistake, and that correction becomes a new episode (--strategy.type=dagger); it’s the most efficient way to improve after the first version.

Why learn this

Without a numerical evaluation, every AI project "almost works." The spreadsheet shows you exactly where you stand, lets you compare versions, and gives you something concrete to present in your final project.

Key concepts

ACT learns a task and ignores text. A VLA (Vision-Language-Action model, introduced in 5.1) also receives the written instruction and can perform different tasks depending on the request: "put the cube in the box" or "put the ball in the box." These models are already pretrained on many robots; you just fine-tune them fine-tune with your episodes.

How to read this map: the difference between a VLA and ACT is the cyan box. With the text, the same model chooses which object to pick up. The text doesn’t become code or a list of steps: it directly conditions the joint movements.

Goal: fine-tune SmolVLA from the base model lerobot/smolvla_base using your dataset. First, record episodes with different instructions (for example, half with "cube" and half with "ball").

# Install the SmolVLA extra in the same LeRobot environment
pip install "lerobot[smolvla]"

# Fine-tune from the base model (check the current docs for the flags)
lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=<seu_usuario_hf>/<dataset> \
  --policy.device=cuda \
  --output_dir=outputs/train/smolvla-<dataset> \
  --batch_size=16 \
  --steps=20000 \
  --wandb.enable=false \
  --policy.push_to_hub=false

Why learn this

The final project (5.6) needs an arm that follows different requests. With ACT, you train one policy per task and the language model chooses which one to call; with a VLA, a single policy receives the request as text. Knowing both approaches lets you choose based on the hardware you have.

Key concepts