PTENES
MODULE 2.5

🎓 Training: Imitation Learning, ACT, Diffusion Policy

How robots learn by watching humans: from demonstrations to neural policies. Behavior cloning, Action Chunking Transformers, and Diffusion Policy are the 3 dominant paradigms—each with clear trade-offs in complexity, data, and performance.

6
Topics
~45
Minutes
Technical
Level
Practical
Type
Imitation Learning Pipeline 🧑‍🏫 Demonstrations pairs (s, a) Behavior Cloning supervised regression ACT (Chunking) Transformer + CVAE Diffusion Policy iterative denoising Neural Policy π(a | o, lang) obs → action 🤖 Deploy

Illustrative diagram — three imitation learning paradigms converging on a deployable neural policy.

1

🎯 Imitation Learning: basic behavior cloning

Behavior cloning (BC) is the simplest form of imitation learning: human demonstrations (observation-action pairs) are collected, and a supervised neural network is trained to map observations to actions. It is essentially regression on trajectories, without any environment simulation.

🔑 Mathematical formulation

# Behavior Cloning objective

π* = argminπ E(s,a)~D [ L(π(s), a) ]

# where D is the human demonstration dataset

# L is MSE for continuous actions, cross-entropy for discrete ones

# Training in LeRobot (ACT as a BC baseline)

python lerobot/scripts/train.py \

policy=act \

dataset_repo_id=lerobot/aloha_sim_insertion_human \

training.num_steps=100000

✓ When BC works well

  • ✓Tasks with low variance — pick up an object always in the same place
  • ✓Many demonstrations — 500+ trajectories covering the state space
  • ✓Rapid prototype: trains on minutes, without reward setup
  • ✓Interpretable baseline for comparison with ACT and Diffusion

✗ Classic BC problems

  • ✗Compounding error — small errors accumulate outside the training distribution
  • ✗Covariate shift — test state ≠ training state, abrupt degradation
  • ✗Poorly captured multimodality—MSE averages across modes, producing an invalid action
  • ✗Requires consistent demos — an inconsistent human ruins the training

💡 Practical tip

Use BC as a mandatory baseline. If BC with 200 demos does not reach >50% success rate, your task is probably multimodal or your dataset has inconsistencies — and ACT/Diffusion Policy will address this systematically.

2

⚡ ACT: Action Chunking with Transformers

ACT (Zhao et al., 2023) predicts chunks of future actions at once, instead of one action at a time. A CVAE encoder generates a latent style, and a transformer decoder generates K future actions conditioned on that style and the current observation—resolving compounding error and multimodality simultaneously.

🏗️ ACT architecture in detail

1. CVAE Encoder

Receives a sequence of human actions → produces z (latent style). At inference, z ~ N(0,1).

2. Visual Backbone

ResNet-18 processes camera images → visual tokens for the decoder.

3. Transformer Decoder

Generates action chunks of K=100. Temporal ensembling combines overlapping chunks.

# ACT configuration in LeRobot

policy.chunk_size = 100 # K actions per inference

policy.n_action_steps = 100

policy.temporal_ensemble_coeff = 0.01

policy.kl_weight = 10 # KL weight in the ELBO

📅 Evolution: from BC to ACT

BC

Behavior Cloning (2010s)

One action per observation. Simple, but with exponentially compounding error over long trajectories.

DAgger

DAgger (Ross et al., 2011)

Iterates by collecting more demos in the states where the agent makes mistakes. Resolves covariate shift but requires an online expert.

ACT

ACT — Action Chunking (Zhao et al., 2023)

Predicts K=100 future actions + CVAE for multimodality. Bimanual with ALOHA: 80%+ success on precise insertion with 50 demos.

💡 Practical tip

ACT is LeRobot's default policy and the right choice for precision manipulation with 50-200 demos. Start with chunk_size=100 e kl_weight=10 — adjust only if the KL loss collapses.

3

🌀 Diffusion Policy: denoising for actions

Diffusion Policy (Chi et al., 2023) applies diffusion models to generating robotic trajectories. It starts with Gaussian noise and, over T denoising steps conditioned on the observation, generates a smooth, multimodal action trajectory — capturing complex distributions that MSE cannot represent.

Diffusion Policy — Denoising Process (T steps) 🎲 Noise N(0, I) t=T ε_θ(aₜ|o) denoising step T→T-1 … ε_θ(aₜ|o) denoising step 2→1 t=0 📈 Trajectory clean + smooth Execution on Robot 🤖 receding horizon Observation o (image + proprioception) conditions every step

Illustrative diagram — Diffusion Policy iterates through T denoising steps conditioned on the current observation.

Comparison: U-Net vs. Transformer backbone

CNN (U-Net)

  • • Faster inference (10-25ms)
  • • Better for low-dimensional actions
  • • 50-100 DDPM steps or 15 DDIM steps
  • • Default choice for single-camera tasks

Transformer

  • • Better scaling with more data and cameras
  • • Long context: history of observations
  • • Higher latency (30-100ms)
  • • Choice for complex multimodal systems

💡 Practical tip

Use num_diffusion_iters=15 with DDIM (instead of 100 steps with DDPM) for real-time inference. The quality loss is minimal, and you keep latency below 50ms for control at 20Hz.

4

🔧 Fine-tuning pretrained VLAs

Fine-tuning adapts a VLA model pretrained on general data (e.g.: OpenVLA trained on Open X-Embodiment with 970k trajectories) for your specific task and robot using just 50-200 demonstrations. This is the practical path for 99% of real-world projects.

✓ Efficient strategies

  • ✓LoRA/QLoRA — fine-tune only 0.1-1% of parameters without degrading the backbone
  • ✓Dataset mixing — mix specific data (70%) with general data (30%) to prevent catastrophic forgetting
  • ✓Cosine decay with warmup — 5% of warmup steps, final LR 1e-5
  • ✓OpenVLA-OFT outperforms RT-2-X with 10× less data

✗ Common errors

  • ✗Full fine-tune with high LR — destroys general features learned during pretraining
  • ✗No dataset mixing — catastrophic forgetting of generalization
  • ✗Evaluate only on the training dataset — overfitting stays invisible without holdout tasks
  • ✗Ignore the base model's image resolution — changing the resolution breaks the ViT

💻 OpenVLA fine-tuning with LoRA

# Install dependencies

pip install openvla transformers peft

# Configure LoRA rank=32 for the action head

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(

r=32,

lora_alpha=64,

target_modules=["q_proj", "v_proj"],

lora_dropout=0.05,

)

# Low LR to avoid catastrophic forgetting

optimizer = AdamW(params, lr=2e-5)

scheduler = cosine_with_warmup(optimizer, warmup=500)

5

🚀 RL post-training: +24% with reinforcement

Reinforcement Learning post-training applies RL (PPO, GRPO) to a policy initialized with imitation learning. The robot improves beyond human demos when optimizing automatic reward signals — such as success detection by camera or force sensor.

📅 IL → RL pipeline

①

Pretraining via Imitation Learning

50-200 human demos → BC or ACT policy as initialization. Ensures safe, near-optimal behavior before RL.

②

Define reward function

Sparse success reward (0/1) + auxiliary dense reward (distance to target position). Reward shaping is critical for sample efficiency.

③

RL fine-tuning with PPO/GRPO

KL penalty vs. the IL policy (to prevent divergence). 10k-100k simulation rollouts. +24% success rate in connector insertion (Physical Intelligence, 2024).

④

Evaluation and deployment

Test in holdout scenarios with variations (position, lighting, objects) before transferring to the real robot.

Key RL post-training concepts

KL penalty

Keeps π_RL close to π_IL, preventing catastrophic collapse. Typical β_KL: 0.01-0.1.

Sample efficiency

RL directly on a real robot: extremely costly. Simulation + sim-to-real is essential for RL at scale.

Online vs Offline RL

Offline RL (IQL, TD3+BC) uses static datasets—safer. Online RL collects new data but requires a fast simulator.

GRPO for VLAs

Group Relative Policy Optimization—a PPO variant with baseline estimation from a group of rollouts. Fewer hyperparameters.

6

📊 Evaluation: success rate, OOD generalization

VLA robotics evaluation measures success rate (% of tasks completed) in in-distribution and out-of-distribution scenarios. Benchmarks like LIBERO, RLBench, and SIMPLER standardize comparisons — essential for knowing whether your policy generalizes or just memorizes the dataset.

🏋️ LIBERO

4 suites with 10 tasks each. Measures spatial, object, goal, and long-horizon generalization. HuggingFace standard.

🤖 RLBench

100 unique tasks with varying difficulty. Strong baseline for comparing imitation learning algorithms.

📐 SIMPLER

Realistic benchmarks with aligned sim-to-real. Measures robustness to variations in lighting, texture, and position.

✓ Rigorous evaluation

  • ✓N≥50 episodes per condition for reliable confidence intervals
  • ✓Test with different objects, positions, and lighting not seen during training
  • ✓Report the mean ± standard deviation across multiple seeds
  • ✓Analyze failure mode: exactly where the robot fails

✗ Poor performance

  • ✗N<10 episodes — masked variance; success count without statistical significance
  • ✗Test only in-distribution — training success isn't real-world success
  • ✗Compare models with different numbers of demonstrations without controlling the data budget
  • ✗Evaluate on a task different from the benchmark’s original published task

💡 Practical tip

Before running 100k training steps, evaluate with n_eval_episodes=20 every 10k steps to detect a plateau early. A flat learning curve after 30k steps with BC usually indicates a multimodal task — switch to Diffusion Policy.

✅ Module Summary

✓
BC → ACT → Diffusion Policy — progression in complexity with real gains: each paradigm solves a problem from the previous one.
✓
Action chunking solves compounding error — predicting K=100 future actions reduces the decision frequency and isolates local errors.
✓
LoRA fine-tuning is sufficient — adapts a pretrained VLA to your task with 50-200 demos and no GPU cluster.
✓
Rigorous evaluation with OOD — in-distribution success rate doesn't predict real-world performance; test with unseen variations.

Next module

2.6 — Simulators: Isaac Sim, MuJoCo, Gazebo, and Sim-to-Real Transfer

← Back to the Learning Path Next module →