From zero to deploy: Python, LeRobot, simulators, ROS2, training, and sim-to-real transfer. The tools and techniques for building VLAs that work.
6 modules · ~18 estimated hours · Prerequisite: Track 1
PyTorch, Transformers, Gymnasium
Open-source collection, training, and deployment
MuJoCo, Isaac Sim, PyBullet
Pub/sub, TF2, Nav2, and VLA
From demonstrations to neural policy
From virtual to physical without breaking
The tools every VLA engineer needs to master: from tensor to environment. PyTorch for autograd, HuggingFace for pre-trained models, Gymnasium for standardized environments.
PyTorch is the dominant deep learning framework in robotics and VLA. Tensors are multidimensional arrays with native GPU support via CUDA. The autograd system computes gradients automatically, which is essential for backpropagation during policy training.
Practically every VLA (OpenVLA, pi-0, ACT) is implemented in PyTorch. Without mastering tensors, shapes, and autograd, you can't debug training, customize losses, or adapt existing architectures.
torch.Tensor, device (cpu/cuda), dtype (float32/bfloat16), autograd and computational graph, nn.Module and forward(), optim.AdamW, mixed precision (torch.amp), torch.compile for JIT optimization.
Library that unifies access to 300k+ pretrained models via a standardized API. For VLA, this is crucial because models such as OpenVLA, SmolVLA, and Qwen-VL are distributed via HuggingFace Hub with a ready-to-use inference pipeline.
The HuggingFace Hub is the center of the open-source VLA ecosystem. LeRobot hosts 58k+ datasets there. Knowing how to use AutoModel, from_pretrained(), and tokenizers is a prerequisite for any hands-on work.
AutoModelForVision2Seq, AutoProcessor, from_pretrained(), model.generate(), LoRA/PEFT for efficient fine-tuning, datasets library, model cards, safetensors format, Hub API.
Gymnasium (successor to OpenAI Gym) defines the standard interface for RL environments: observation_space, action_space, step(), reset(). In VLA robotics, it serves as a wrapper for simulators and lets you train policies with a consistent interface.
The Gymnasium API is the universal contract between policy and environment. MuJoCo, Isaac Sim, and PyBullet all implement this interface. Understanding obs/act/reward/done is fundamental to any training loop.
gym.Env, observation_space (Box, Dict), action_space (continuous Box, Discrete), step() returns (obs, reward, terminated, truncated, info), VectorEnv for parallelism, wrappers for transformations.
Robotics data consists of trajectories: sequences of (observation, action, reward) collected through teleoperation or simulation. The RLDS format (Google) and LeRobot Dataset standardize how to store and load these trajectories for efficient training.
Data is the number one bottleneck in robotics. Knowing how to structure trajectory datasets, apply data augmentation (crop, color jitter), and use DataLoaders with prefetch is essential for training that converges.
RLDS format (TFRecord), LeRobot Dataset (Parquet + video), torch.utils.data.DataLoader, num_workers, pin_memory, collate_fn for batches of variable-length trajectories, robotics-specific data augmentation.
W&B is the standard platform for tracking ML experiments. It records metrics (loss, success rate), hyperparameters, and artifacts (checkpoints), and lets you compare runs side by side. LeRobot has native W&B integration.
Without structured tracking, VLA training becomes a black box. You need to compare ACT vs. Diffusion Policy, different learning rates, and identify when the model diverges. W&B makes this visual and reproducible.
wandb.init(), wandb.log(), wandb.config, Sweeps for hyperparameter search, Tables to visualize trajectories, model registry for versioning, LeRobot integration via YAML config.
Docker containerizes the development environment: exact versions of Python, PyTorch, CUDA, and dependencies stay fixed. In robotics, where the CUDA driver + PyTorch + MuJoCo combination can easily break, containers are essential.
Reproducibility is one of the biggest challenges in robotics research. A Dockerfile ensures anyone can run your training. NVIDIA NGC and LeRobot provide ready-to-use containers.
Multi-stage Dockerfile, nvidia-docker2 (GPU passthrough), NVIDIA NGC containers, docker-compose for multiple services, volumes for datasets, devcontainers for VS Code, CI/CD with containers.
HuggingFace’s open-source framework that democratizes robotics: data collection, training, and deployment. With 12k stars on GitHub and 58k+ datasets, LeRobot is the most accessible entry point for practical VLA.
LeRobot is HuggingFace's open-source framework for end-to-end robotics. Launched in 2024, it already has 12k stars on GitHub and 58k+ datasets on the Hub. It covers data collection, policy training, and deployment on real robots with a single tool.
It’s the fastest path from zero to a working robot. Before LeRobot, every lab had its own ad hoc pipeline. Now there’s an open-source standard with an active community and extensive documentation.
pip install lerobot, lerobot/scripts/train.py, lerobot/scripts/record.py, LeRobotDataset class, HuggingFace Hub integration, open-contribution model, support for multiple robots (SO-100, Koch, Aloha).
Teleoperation is when a human remotely controls the robot to generate demonstrations. LeRobot synchronously records video from the cameras and actuator positions. Each demonstration becomes an episode in the dataset with aligned timestamps.
The quality and quantity of demonstration data directly determine training success. Knowing how to set up teleoperation, calibrate cameras, and record consistent episodes is the foundation for everything.
Leader-follower setup, intrinsic/extrinsic camera calibration, episode recording, lerobot record --robot-path, 50-100 episodes for simple tasks, data quality filtering, replay for validation.
Policies are neural models that map observations to actions. LeRobot implements several: ACT (Action Chunking with Transformers) uses a CVAE + transformer, Diffusion Policy uses iterative denoising, and TDMPC combines model-based planning with learned dynamics.
Each policy has tradeoffs: ACT is quick to train but can be imprecise on fine-grained tasks. Diffusion Policy generates smoother actions but is slower at inference. Knowing how to choose the right policy for the task is crucial.
ACT: CVAE encoder + transformer decoder, action chunks of 100 steps. Diffusion Policy: DDPM with U-Net or transformer, 10-100 denoising steps. TDMPC: world model + planning. VQ-BeT: discrete action tokenization.
LeRobot uses YAML configs to define the entire training pipeline: policy, dataset, hyperparameters, device. A single command (python lerobot/scripts/train.py) launches training with automatic logging to W&B and periodic checkpoints.
The YAML config is the experiment’s “contract.” Knowing how to adjust the learning rate, batch size, number of epochs, and checkpoint scheduling is the difference between a model that converges and one that diverges.
Hydra config system, policy.yaml + env.yaml, training_steps, eval_freq, checkpoint_freq, resume_from_checkpoint, GPU memory management (gradient accumulation, mixed precision), multi-GPU with DDP.
Robotics evaluation differs from NLP: measuring loss on the validation set isn’t enough. What matters is success rate (% of episodes in which the task was completed) and generalization to new objects and positions. LeRobot integrates automated evaluation in simulators.
A model with low loss can fail on a real robot. Simulation success metrics are the most reliable proxy before deployment. Knowing how to configure periodic evaluation during training speeds up iterations.
Success rate, partial completion score, out-of-distribution (OOD) evaluation, sim evaluation via gym env, N_eval_episodes (typically 50), video recording of episodes, benchmark comparisons (SIMPLER, LIBERO).
Deployment means transferring the trained checkpoint to control a real robot in real time. LeRobot supports accessible robots such as SO-100 (~$300) and Koch v1.1, running inference on the laptop itself. The loop is: capture a frame, infer an action, send a command to the actuator.
Real-world deployment is where everything is truly put to the test. Camera latency, USB bandwidth, operating system jitter—all of it matters. Knowing how to diagnose deployment issues is the skill that separates simulation from real robotics.
lerobot/scripts/control_robot.py, real-time inference loop, camera latency budgeting, USB serial communication, safety limits (maximum speed, force threshold), post-deploy calibration, rapid collect-train-deploy iteration.
Train robots without breaking anything: high-fidelity physics simulation for VLA. MuJoCo at 2.7M steps/s, Isaac Sim with ray tracing at 94k FPS, and PyBullet for rapid prototyping.
MuJoCo (Multi-Joint dynamics with Contact) is a physics engine optimized for simulating contact between rigid bodies and joints. Acquired by Google DeepMind in 2021, it is now open-source and processes 2.7 million steps per second on a CPU.
MuJoCo is the standard simulator in the robotics and RL academic community. Most benchmarks (dm_control, Gymnasium Robotics) run on MuJoCo. Mastering MuJoCo is a prerequisite for reproducing papers and training VLA policies at scale.
MJCF (XML format for models), qpos/qvel (generalized state), Newton/CG solver, soft contacts, semi-implicit Euler integrator, mujoco-py vs. mujoco (official binding), MJX (MuJoCo in JAX for GPU).
Isaac Sim is NVIDIA’s robotics simulation platform, built on Omniverse and PhysX 5. It offers photorealistic rendering with real-time ray tracing and can simulate thousands of robots in parallel using GPUs, reaching 94k FPS.
For high-fidelity sim-to-real, the simulator’s visual quality matters a lot. Isaac Sim enables domain randomization with realistic textures and lighting, reducing the reality gap. It is the simulator NVIDIA chose to train GR00T N1.
Omniverse USD (universal scene format), PhysX 5 (GPU-accelerated physics), Isaac Lab (RL/IL framework), OmniGraph (visual logic), RTX ray tracing, Isaac Gym (predecessor, deprecated), integration with ROS2 via bridge.
PyBullet is the Python wrapper for the Bullet Physics Engine. With a simple pip install, you get a 3D simulator with collision detection, constraints, and a URDF loader. Ideal for rapid prototyping and teaching.
The barrier to entry is minimal: it runs on any laptop without a dedicated GPU. Many RL tutorials and courses use PyBullet. It’s the fastest way to get a simulated robot running a learned policy.
URDF (Unified Robot Description Format), p.loadURDF(), stepSimulation(), getJointState/setJointMotorControl2, pybullet_envs (Gymnasium wrappers), direct vs GUI mode, collision shapes, constraints.
Gazebo (now "Gz") is the simulator integrated into the ROS2 ecosystem. Unlike MuJoCo (focused on RL research), Gazebo simulates real sensors (LiDAR, depth cameras, IMU) and communicates with ROS2 natively via topics.
If the final deployment uses ROS2, testing in Gazebo means the real robot and simulated robot code are virtually identical. This greatly speeds up the development cycle and reduces surprises during deployment.
SDF (Simulation Description Format), gz-sim (new name), ros_gz_bridge, sensor plugins (camera, LiDAR, IMU), world files, spawn models via service, Gazebo Fuel (3D model repository), MoveIt2 integration.
Domain randomization is the technique of randomly varying simulator parameters (textures, lighting, masses, friction, colors) during training so the policy learns to be robust to variations and transfers better to the real world.
Without domain randomization, policies trained in simulation fail catastrophically in the real world because they overfit to the simulator's visuals and perfect physics. It's the technique that enabled the first successful sim-to-real transfer (OpenAI Rubik's Cube, 2019).
Visual randomization (textures, colors, light), physics randomization (mass, friction, damping), dynamics randomization, structured vs. uniform sampling, randomization curriculum (progressive), ADR (Automatic Domain Randomization).
Sim-to-sim is the practice of transferring a policy trained in one simulator (e.g., MuJoCo) to another simulator (e.g., Isaac Sim) as an intermediate step before using a real robot. If the policy fails in another simulator, it will fail in the real world.
It’s a cheap, fast validation step. Testing in 2+ simulators with different physics reveals policy weaknesses before risking expensive hardware. It also helps isolate whether a problem is visual or related to dynamics.
Cross-simulator transfer, physics fidelity gap between simulators, URDF/MJCF conversion tools, benchmark standardization (SIMPLER), statistical evaluation (N episodes, confidence intervals), visual vs. dynamic debugging.
The operating system for robots: pub/sub, TF2, Nav2, and VLA integration. ROS2 is the industry standard for production robotics.
ROS2 is the standard middleware framework in the robotics industry. Unlike ROS1, it uses DDS (Data Distribution Service) as its transport layer, providing configurable QoS, automatic node discovery, and real-time communication without a central master node.
Any robot going into industrial production uses ROS2. It’s the “Linux of robotics.” Integrating a VLA model into ROS2 is required for real-world deployment. Current distributions (Humble LTS, Iron) are stable and supported through 2027+.
DDS (Fast-RTPS, Cyclone DDS), nodes, executors (single-threaded vs. multi-threaded), lifecycle nodes, QoS profiles (reliability, durability, history), colcon build system, ament_cmake/ament_python.
The three communication patterns in ROS2. Topics are asynchronous pub/sub (data streaming, e.g., camera images). Services are synchronous request/reply (e.g., changing modes). Actions are long-running tasks with feedback (e.g., moving an arm to a position).
Choosing the wrong pattern causes subtle bugs: using a topic where an action should be means losing progress feedback; using a service where a topic should be blocks the node. VLA inference typically publishes to a topic (high frequency) and receives commands via an action.
Publisher/Subscriber (topic), Client/Server (service), ActionServer/ActionClient, msg/srv/action interfaces, custom messages, QoS for each type, ros2 topic echo/pub for debugging, rqt_graph for visualization.
TF2 (Transform Library 2) maintains a tree of 3D transformations between all of the robot’s reference frames (base_link, camera, gripper, etc.), updated in real time. It lets you convert coordinates between any pair of frames instantly.
VLA needs to know where the camera is relative to the gripper and the world. Without TF2, you would have to calculate transformations manually for every frame. TF2 does this automatically with a temporal buffer, which is essential for sensor fusion.
Frames (base_link, odom, map, camera_link, tool0), static vs. dynamic transforms, tf2_ros::Buffer, lookupTransform(), URDF → robot_state_publisher → TF tree, tf2_echo for debugging, rviz2 for visualization.
Nav2 is the autonomous navigation stack (path planning, obstacle avoidance, costmaps). MoveIt2 is the manipulation framework (motion planning, inverse kinematics, collision avoidance). Together, they cover mobility and manipulation.
VLA can generate high-level actions ("go to the table"), but Nav2/MoveIt2 execute safe path planning. The combination of VLA (what to do) + Nav2/MoveIt2 (how to get there without colliding) is the standard for mobile manipulator robots.
Nav2: BT (Behavior Trees), costmap2d, planner/controller servers, recovery behaviors. MoveIt2: SRDF, planning scene, OMPL planners (RRT*, PRM), servo (real-time control), grasp planning, pick-and-place pipeline.
The VLA inference node in ROS2 receives images and language commands via topics, runs the VLA model (PyTorch/ONNX), and publishes actions to the robot controller. It acts as a bridge between the ROS2 world and the ML world.
Integrating ML into ROS2 has unique challenges: Python GIL, serialization latency, camera-action synchronization, GPU memory management. A well-architected node is the difference between 5 Hz and 50 Hz inference.
rclpy node with image callback, message_filters (synchronize camera + joint_states), ONNX Runtime for optimized inference, TensorRT via ROS2, pre-allocated buffers (avoid malloc), multi-threaded executor, composable nodes.
Launch files in ROS2 (Python) orchestrate the startup of multiple nodes with parameters, remappings, and conditionals. In production, they work with systemd, Docker, and monitoring to ensure the robotic system restarts automatically after failures.
Deploying robotics systems in production is systems engineering, not just ML. A well-structured launch file with lifecycle management, health checks, and restart policies is what separates a demo from a product. Companies pay a premium for this expertise.
LaunchDescription, Node(), ComposableNodeContainer, lifecycle_node (configure/activate/deactivate), parameters YAML, launch arguments, event handlers, ros2 launch logging, systemd unit files, Docker + ROS2, rosbag2 for replay.
How robots learn by watching humans: from demonstrations to neural policies. Behavior cloning, Action Chunking Transformers, and Diffusion Policy are the 3 dominant paradigms.
Behavior cloning (BC) is the simplest form of imitation learning: human demonstrations (observation-action pairs) are collected, and a supervised neural network is trained to map observations to actions. It is essentially regression over trajectories.
BC is the baseline for all VLA robotics. Although simple, it works surprisingly well with enough data. OpenVLA, RT-2, and Octo use sophisticated BC variants as a starting point. Understanding its limitations (distribution shift) is essential.
Supervised learning on (s,a) pairs, MSE/L1 loss for continuous actions, distribution shift (compounding errors), DAgger (Dataset Aggregation) to mitigate drift, covariate shift, demonstration horizon vs. execution horizon.
ACT (Zhao et al., 2023) predicts chunks of future actions all at once, rather than one action at a time. A CVAE encoder generates a latent style, and a transformer decoder generates K future actions conditioned on that style and the current observation.
ACT addresses two problems in BC: temporal compounding error (by planning K steps ahead) and multimodality (via CVAE). It's LeRobot's default policy and the foundation of ALOHA (Google's bimanual arms). State-of-the-art performance with few demos.
Action chunking: predict K actions at once (K=100 typical). CVAE (Conditional Variational Autoencoder) to capture multiple modes. Temporal ensemble: weighted average of overlapping chunks. kl_weight (latent regularization).
Diffusion Policy (Chi et al., 2023) applies diffusion models (like those used for image generation) to generating robotic trajectories. It starts with Gaussian noise and, over T denoising steps conditioned on the observation, generates a smooth, multimodal action trajectory.
Diffusion Policy outperforms BC and ACT on highly multimodal tasks (e.g., multiple valid paths to the same goal). It is the foundation of GR00T N1’s S1 policy and π0. Its ability to capture complex action distributions is unmatched.
DDPM (Denoising Diffusion Probabilistic Models) applied to actions, noise schedule, conditional denoising (observation-conditioned), U-Net vs. transformer backbone, inference steps (DDIM to speed things up), action horizon, observation horizon vs. action horizon.
Fine-tuning adapts a VLA model pretrained on general data (e.g.: OpenVLA trained on Open X-Embodiment) to your specific task and robot using a much smaller dataset. Similar to fine-tuning LLMs, but with robot trajectories.
Training a VLA from scratch requires millions of trajectories. Fine-tuning a pretrained model takes just 50-200 demonstrations from your setup. Fine-tuned OpenVLA 7B outperforms RT-2-X with 10x less data. It’s the practical path for 99% of projects.
LoRA / QLoRA for memory efficiency, learning rate schedule (cosine decay with warmup), frozen backbone vs. full fine-tuning, dataset mixing (general + specific data), catastrophic forgetting, evaluation on holdout tasks.
Reinforcement Learning post-training applies RL (PPO, GRPO) to a policy initialized with imitation learning. The robot improves beyond human demos by optimizing automatic reward signals. Analogy: RLHF for LLMs, but with physical rewards.
Qwen-VLA showed a +24% improvement in success rate after RL post-training with GRPO. The model outperforms the original human demonstrations. This is the current performance frontier: IL for bootstrapping, RL for optimization.
GRPO (Group Relative Policy Optimization), PPO (Proximal Policy Optimization), reward shaping for manipulation, success-based sparse reward vs. dense reward, KL penalty (not diverging too far from IL), sample efficiency, online vs. offline RL.
VLA robotics evaluation measures success rate (% of tasks completed) in in-distribution scenarios (same objects as in training) and out-of-distribution scenarios (new objects, positions, and lighting). OOD success rate is the metric that really matters.
A model with 95% in-distribution performance may have 30% OOD performance. Knowing how to design evaluations that test real generalization prevents surprises at deployment. Benchmarks such as SIMPLER and LIBERO standardize comparisons across models.
Success rate (SR), OOD SR, partial completion metrics, N_eval_episodes (50+), confidence intervals, SIMPLER benchmark, LIBERO suite, video recording of failures, error taxonomy (grasp fail, reach fail, collision), ablation studies.
The toughest bridge: how to transfer skills from the simulator to the real robot. Domain randomization, system identification, and the real-to-sim-to-real loop.
The reality gap is the difference between a robot's behavior in a simulator and in the real world. It comes from three sources: visual (textures, lighting), dynamics (friction, masses, compliance), and sensing (camera noise, latency, jitter). A perfect policy in simulation can fail 100% of the time in the real world.
Ignoring the reality gap is the most common beginner mistake. Understanding its sources helps you choose the right techniques: domain randomization for the visual gap, system identification for the dynamics gap, sensor noise injection for the sensory gap.
Visual gap (rendering vs. real camera), dynamics gap (simplified physics vs. real contact), sensor gap (ideal vs. noisy sensors), actuator gap (perfect control vs. backlash and friction), transferability metrics, sim-to-real success rate.
Domain randomization systematically varies simulator parameters (textures, colors, light positions, masses, friction coefficients) in each episode. The policy is forced to generalize because it never sees the same scenario twice.
It’s the most robust and widely used sim-to-real technique. OpenAI used it to solve Rubik's Cube (2019), NVIDIA uses it in Isaac Lab, and every modern VLA pipeline includes some form of randomization. Without it, policies overfit to the sim’s appearance.
Visual DR: random textures, background color, light intensity, camera position. Physics DR: mass ±30%, friction ±50%, damping, action delay (1-3 steps). ADR (Automatic Domain Randomization): automatically increases ranges when performance stabilizes.
System identification (sysid) is the process of measuring the robot’s actual physical parameters (masses, inertias, friction, stiffness) and configuring the simulator to match them. Instead of randomizing everything, you calibrate the simulation to be as close to reality as possible.
Domain randomization with very wide ranges can degrade performance (the policy becomes "mediocre at everything"). System identification lets you use narrow ranges centered on the real values, combining the best of both worlds: robustness + precision.
Parameter estimation: record a real trajectory, optimize the sim parameters to minimize error. Bayesian optimization for sysid. Residual physics learning: a neural network that learns the difference between sim and reality. Sim calibration tools (MuJoCo XML editing, Isaac Lab config).
Sim-to-real transfer is the complete pipeline for taking a policy trained in simulation and making it work on a real robot. It includes: training with DR/sysid in simulation, sim-to-sim validation, initial deployment with safety constraints, iteration with real-world data.
It’s the bottleneck for all learning-based robotics. Mastering sim-to-real is the most valued skill in the industry because it determines whether months of training in sim translate into real-world performance or wasted GPU hours.
Progressive deployment: sim → sim-to-sim → real with safety limits → full real. Action smoothing (low-pass filter on commands). Safety constraints (force limits, velocity limits, workspace bounds). Online adaptation: adjust the policy with real-world data.
Real-to-sim-to-real is an iterative loop: collect data on the real robot, use that data to calibrate/improve the simulator, retrain the policy in the improved sim, and deploy again. Each iteration improves both the sim and the policy.
A single sim-to-real pass is rarely enough. The iterative loop is how commercial robots are developed in practice. Figure, NVIDIA, and Physical Intelligence run dozens of real-to-sim-to-real iterations before declaring a behavior "production-ready."
Data flywheel: more real data → better sim → better policy → more real data. Automatic sim tuning with real data (neural sim). Digital twin: exact replica of the real setup in the sim. Continuous deployment pipeline: CI/CD for robotic policies.
Sim-to-real milestones: OpenAI solved the Rubik’s Cube with a Dexterous robotic hand (2019) using massive ADR. NVIDIA trains humanoid locomotion in Isaac Sim with 4096 parallel environments. Google RT-2 used sim-to-real to validate manipulation tasks.
These cases show that sim-to-real works for extremely complex tasks. The lessons learned (randomization scale, the importance of tactile sensors, reward design) apply directly to any VLA project.
OpenAI Rubik's: ADR with 200+ randomized parameters, 13K years of sim experience. NVIDIA humanoid: massively parallel sim, PPO with 4096 envs, 45 min wall-clock. Google RT-2: SIMPLER benchmark for sim validation. Lesson: simulation scale pays off.