🗄️ Datasets: Open X-Embodiment and DROID
The data that powers VLAs — scale, diversity, and format. From millions of robot trajectories to the RLDS pipeline that transforms raw collection into foundation models.
🌐 Open X-Embodiment: the dataset that changed everything
Open X-Embodiment (OXE) is the largest collaborative robotics data effort ever undertaken. More than 1 million trajectories collected by 21 institutions, covering 22 different embodiments—from industrial arms to bipedal robots. It’s the foundation that made generalist VLAs possible.
Why OXE is a historic milestone
Franka, UR, Kuka, Spot, Hello Robot, and more — diverse bodies in a single dataset.
Google, Stanford, Berkeley, CMU, and others collaborating on an open dataset.
Each trajectory is a sequence of steps with an observation, action, and language metadata.
✓ What OXE enables
- ✓ Cross-embodiment pretraining (RT-X, OpenVLA)
- ✓ Transfer learning across robots from different families
- ✓ Reproducible research with a common baseline
- ✓ Study of scaling laws for robotics
✗ Real OXE limitations
- ✗ Uneven distribution: some robots have 10x more data
- ✗ Controlled laboratory environments — little in-the-wild testing
- ✗ Quality varies across contributors
- ✗ Inconsistent language annotations across institutions
Training on a multi-robot dataset lets the model learn policies that transfer across different bodies.
A family of Google models trained on OXE; showed that collaborative data outperforms isolated proprietary data.
Episodic sequence of (observation, action, reward) that records how a robot performed a task.
Empirical pattern: more diverse trajectories → better generalization. But diversity matters more than raw volume.
🏙️ DROID: in-the-wild diversity
The DROID (Distributed Robot Interaction Dataset) offers something OXE doesn’t: uncontrolled real-world environments. With 76 thousand trajectories across 564 distinct scenes—offices, kitchens, bathrooms—and a single standardized piece of hardware (Franka arm), DROID is the largest openly available in-the-wild manipulation dataset.
📊 DROID numbers
💡 Why a single robot is an advantage
Using only the Franka eliminates hardware variance. The only diversity factor is the environment — exactly what we want to study. The model learns to generalize to new spaces, not new bodies. This is the right abstraction for environmental robustness research.
✓ DROID excels at
- ✓ Robustness to lighting and layout variations
- ✓ Deployment in non-laboratory environments
- ✓ Fast fine-tuning for new spaces
✗ DROID limitations
- ✗ Only 86 tasks — low semantic diversity
- ✗ Single-arm only — no bimanual control or locomotion
- ✗ Does not generalize across embodiments on its own
Data collected in real, uncontrolled environments, with objects and scenarios that were not prepared — the opposite of a lab setting.
Coordinated collection across multiple sites using the same hardware and protocol, ensuring consistency despite diverse environments.
🍳 BridgeData V2: kitchen-scale manipulation
BridgeData V2 is the dataset that proves accessible, reproducible data collection works. With more than 60,000 trajectories collected with a low-cost WidowX arm and focused on kitchen household tasks, it became the standard benchmark for fine-tuning and cross-evaluation among VLAs.
How BridgeData evolved:
BridgeData V1 (2022)
First ~7k trajectories with WidowX. Shows the feasibility of low-cost data collection for household manipulation.
BridgeData V2 (2023)
Scales to 60k+ trajectories, diversifies tasks and kitchen environments. Integrated into OXE as a subset.
Current use as a benchmark
Standard for evaluating VLA transfer, comparing fine-tuned policies, and testing language instruction generalization.
💡 WidowX as proof of concept
The WidowX costs ~$3,000 — a fraction of the cost of industrial arms. BridgeData proves that the barrier to data collection isn’t expensive hardware, but protocol consistency e a task variety. This democratized VLA research beyond the major labs.
A variety of tasks within the same environment provides a rich signal for language-action alignment.
BridgeData is the standard dataset for measuring how well a generalist VLA adapts to new home environments.
Low-cost hardware + consistent protocol = scalable data collection. The robot's cost does not determine the quality of the data.
📦 RLDS / TFDS format
RLDS (Reinforcement Learning Datasets) is the de facto standard for robotic data. Built on TFDS (TensorFlow Datasets), it organizes trajectories as sequences of episodes with structured steps—enabling efficient streaming of datasets with millions of trajectories.
💡 Action normalization: the critical step
Each dataset has its own action space: torque commands, joint velocities, delta position, absolute coordinates. To combine OXE + DROID + BridgeData in a single training run, you need to normalize all actions to the same range. This single step determines whether fine-tuning will converge or diverge.
Complete sequence of steps for a task: from the initial state to completion (success or failure). Basic unit of learning by imitation.
A single timestep: current observation → action taken → reward received → next state. The granularity of learning.
Loads data from disk on demand without needing everything in RAM. Enables training on millions of trajectories with limited hardware.
Standardize names such as image_primary, state, action is required to combine heterogeneous datasets in a single DataLoader.
🎮 Data Collection via teleoperation
Almost all VLA data comes from human teleoperation: one operator controls the robot in real time while observations and actions are recorded. Operator quality, the collection setup, and demonstration density directly affect the learned policy.
Teleoperation methods compared
✓ Data collection best practices
- ✓ Smooth demonstrations, without excessive hesitation
- ✓ Vary the objects' initial positions between episodes
- ✓ Include failure recovery (recovery demos)
- ✓ Multiple operators to cover different styles
✗ Common data collection errors
- ✗ Repeated hesitation "contaminates" the policy with pauses
- ✗ Fixed position → robot learns the position, not the task
- ✗ Ignore failure episodes that could be informative
- ✗ Single operator → movement-style bias
💡 The effect of hesitation on the policy
If the operator hesitates before grasping an object, the model learns that hesitation is part of the policy. The robot will pause before each grasp—even when it doesn't need to. Clean, fluent demonstration data produce more decisive and robust policies than abundant but highly hesitant data.
⚖️ Quality vs. Quantity and Scaling Laws
In NLP, more data = better model is almost always true. In robotics, the relationship is more subtle: diversity and quality outweigh raw volume. Understanding the scaling laws for robotic data is essential to avoid wasting collection resources.
📊 What scaling laws tell us about robotic data
Double the data volume of same environment delivers diminishing marginal gains after a certain point.
Add new task types or new ones environments delivers consistent gains even with few new demos.
Poor-quality data (hesitation, collisions, anomalous trajectories) does more harm than good — filtering is important.
Larger models make better use of large datasets—but small models can saturate with just 10k high-quality demos.
✓ Efficient data strategy
- ✓ Prioritize task diversity over volume
- ✓ Filter episodes with anomalies (collisions, timeouts)
- ✓ Balance datasets when mixing sources (OXE + DROID)
- ✓ Measure the action distribution before training
✗ Pitfalls of large datasets
- ✗ Assume that more demos always = a better policy
- ✗ Mix datasets without normalizing action spaces
- ✗ Ignore the imbalance (90% grasping, 10% releasing)
- ✗ Does not check for correlations between language and action in the data
💡 Emergent capabilities at scale
VLAs trained on the full OXE show capabilities that do not appear in smaller subsets: task composition, generalization to unseen objects, and following ambiguous instructions. These abilities emergent emerge at scale — but only if the source data covers sufficient semantic diversity, not just volume.
Empirical relationship between data volume, model size, and performance. In robotics, task diversity scales better than sheer volume.
A sampling technique that ensures smaller datasets (BridgeData) aren’t eclipsed by larger datasets (OXE) during training.
Abilities that emerge in larger models trained on more data without being explicitly programmed — zero-shot generalization, task composition.
Process of removing anomalous trajectories, collisions, and low-quality demonstrations before training. Has a disproportionate impact on the quality of the final policy.
✅ Module Summary
Next module
1.5 — Hardware: Cobots, Humanoids, and Platforms