Track map
๐ Python + PyTorch
The stack where every VLA is born, trained, and served.
๐ค LeRobot
Record, train, and deploy in a single open repo.
๐ Simulators
Thousands of worlds per night, without breaking anything.
๐ ROS2
The glue between policy and torque, in real time.
๐ฆฟ Hardware
Arms, grippers, sensors, and Jetson at the edge.
๐ฆ Datasets
1M+ trajectories: fuel for the generalist.
Detailed content
๐ Python + PyTorch: The Dominant Stack for VLA
What it is: eager execution, autograd, and the HF ecosystem.
Why it matters: rapid iteration on architectures that change every week.
Concepts: define-by-run, CUDA/MPS, portability.
What it is: bf16/fp16 and 7B memory budget.
Why it matters: defines which GPU trains the model.
Concepts: autocast, Adam states, activations.
What it is: load VLMs and train QLoRA adapters.
Why it matters: fine-tune a 7B model on 24 GB.
Concepts: PEFT, NF4, target_modules.
What it is: Dataset/DataLoader and action normalization.
Why it matters: incorrect normalization breaks the action head.
Concepts: stats with ckpt, augmentation, workers.
What it is: state sharding across GPUs.
Why it matters: makes training large VLAs feasible.
Concepts: all-gather, reduce-scatter, recompute.
What it is: torch.compile and action chunking.
Why it matters: latency closes the control loop.
Concepts: inference_mode, CUDA graphs, export.
๐ค LeRobot: End-to-End Framework
What it is: datasets, policies, sim, and hardware in one repo.
Why it matters: democratizes robotics the way transformers did for NLP.
Concepts: record, train, eval; Hub.
What it is: state in Parquet, images in MP4.
Why it matters: single reproducible format.
Concepts: features, episode_index, delta_timestamps.
What it is: policy classes with the same interface.
Why it matters: switch methods with a flag.
Concepts: chunking, flow matching, checkpoints.
What it is: inexpensive arms and leader-follower teleoperation.
Why it matters: collect demos for ~$100.
Concepts: calibration, recording, consistency.
What it is: three commands + inference loop.
Why it matters: full cycle on a tabletop arm.
Concepts: eval rollout, success rate, closed loop.
What it is: datasets and checkpoints versioned on the Hub.
Why it matters: reproduces papers and contributes demos.
Concepts: push_to_hub, dataset card, visualize.
๐ Simulators: MuJoCo, Isaac, Habitat
What it is: cheap, safe, parallel data.
Why it matters: RL requires many interactions.
Concepts: infinite reset, ground-truth, randomization.
What it is: precise contact, differentiable physics in JAX.
Why it matters: fine contact manipulation.
Concepts: MJCF, vmap, menagerie.
What it is: GPU simulation with RTX and massive-scale RL.
Why it matters: thousands of envs without a CPU bottleneck.
Concepts: PhysX 5, OpenUSD, photorealism.
What it is: navigation and embodied QA in real-world scenes.
Why it matters: spatial perception and exploration.
Concepts: HM3D, EQA, SAPIEN/Genesis.
What it is: physics, sensors, and appearance diverge.
Why it matters: 95% in simulation can be 30% in the real world.
Concepts: reality gap, deformables, honest eval.
What it is: standardized evaluation suites.
Why it matters: compare policies across papers.
Concepts: success rate, RoboSuite, generalization.
๐ ROS2: Middleware for Real-World Robotics
What it is: decentralized middleware over DDS.
Why it matters: real time and safety.
Concepts: nodes, topics, services, actions.
What it is: perception, planning, and control as nodes.
Why it matters: swap the policy without changing the driver.
Concepts: rqt_graph, topic hz, modularity.
What it is: rclpy node that subscribes to camera+state.
Why it matters: observations from the same instant; correct QoS.
Concepts: message_filters, BEST_EFFORT, callback group.
What it is: timed frame tree.
Why it matters: convert camera frame to base.
Concepts: hand-eye, REP-103, timestamp.
What it is: planning, collision, IK/FK, limits.
Why it matters: safe approach and trajectory execution.
Concepts: ros2_control, e-stop, time parameterization.
What it is: frameโtorque budget and jitter.
Why it matters: jitter destabilizes control.
Concepts: PREEMPT_RT, watchdog, chunking.
๐ฆฟ Hardware: Arms, Sensors, Edge
What it is: DOF, payload, and repeatability.
Why it matters: choose the arm based on the task/budget.
Concepts: force control, cobot, ~$100.
What it is: parallel, suction, multi-finger, GelSight.
Why it matters: end-effector defines what gets grasped.
Concepts: dexterity, in-hand, contact feedback.
What it is: RealSense/ZED, wrist cam, encoders.
Why it matters: vision is the dominant input.
Concepts: anti-occlusion, observation.state, F/T.
What it is: on-board compute, throughput vs. power consumption.
Why it matters: eliminates network latency.
Concepts: TensorRT, INT8, thermal envelope.
What it is: leader-follower, VR, gloves.
Why it matters: good data starts with good teleop.
Concepts: action space, retargeting, ergonomics.
What it is: from $100 to a research cell.
Why it matters: cost plummeted; the method matters more.
Concepts: affordable kit, cloud + edge.
๐ฆ Datasets: Open X-Embodiment, DROID
What it is: 1M+ trajectories, 22 embodiments.
Why it matters: the "ImageNet of robotics".
Concepts: 60+ general-purpose datasets, unified corpus.
What it is: co-training across multiple embodiments.
Why it matters: positive transfer between robots.
Concepts: mixing weights, co-fine-tune.
What it is: 76k demos, 564 scenes, Franka.
Why it matters: diversity against overfitting.
Concepts: standardized, 86 tasks, real-world environments.
What it is: transfer benchmark, 4 suites.
Why it matters: measure learning without forgetting.
Concepts: Spatial/Object/Goal/Long, composition.
What it is: common format + heterogeneous action spaces.
Why it matters: without normalization, there is no transfer.
Concepts: TFDS, EEF ฮpose, frequency.
What it is: filter for quality and balance coverage.
Why it matters: quality beats raw volume.
Concepts: embodiment bias, license, contribute.