Detailed content
🔹 Open X-Embodiment: 1M+ trajectories, 22 embodiments
O Open X-Embodiment (OXE, 2023) is the “ImageNet of robotics”: a collaboration of ~20 labs that unified 60+ datasets in 1M+ trajectories through 22 robot bodies different — from table arms to mobile bases. This made it feasible to train generalist VLAs like OpenVLA and Octo.
📊 OXE in numbers
- 1M+ — real demonstration trajectories.
- 22 — distinct embodiments (robot types).
- 60+ — aggregated source datasets.
- 527 — skills covered (approx.).
# OXE é distribuído via TFDS; cada fonte é um dataset RLDS
import tensorflow_datasets as tfds
b = tfds.builder_from_directory("gs://gresearch/robotics/fractal20220817_data/0.1.0")
ds = b.as_dataset(split="train")
# misturar várias fontes com pesos -> treino generalista
sources = ["fractal20220817_data", "bridge", "kuka", "taco_play"] # subset OXE⚡ Practical tip
The full OXE dataset is tens of TB. In practice, you select a mix from relevant sources (the "OXE magic soup" from OpenVLA) instead of downloading everything — what matters is embodiment and task coverage, not raw volume.
OXE
Unified corpus.
Embodiment
Type of robot.
Scale
1M+ trajectories.
Generalist
Based on OpenVLA/Octo.
🔹 RT-X and cross-embodiment co-training
The central finding of the papers RT-X (RT-1-X, RT-2-X) was the positive transfer: training a single model on multiple bodies improves performance on each one compared with training in isolation. Data from a Franka arm helps generalization to a WidowX — knowledge "leaks" across embodiments.
✓ Do
- ✓Co-train across multiple embodiments to generalize.
- ✓Mix large datasets (OXE) with a few demos from your robot.
- ✓Weight (sampling weights) datasets by quality/size.
✗ Avoid
- ✗Assuming positive transfer without normalizing action spaces.
- ✗Let a giant dataset dominate the sampling.
- ✗Ignore embodiment bias (more data ≠ more diverse).
⚡ Practical tip
Positive transfer isn’t automatic: it depends on a shared action representation and mixing weights well chosen. It’s the empirical basis for why generalist VLAs work.
RT-X
Cross-embodiment co-training.
Transfer +
Knowledge leaks.
Mixing weights
Balance datasets.
Co-fine-tune
OXE + your demos.
🔹 DROID: 76k demos, 564 scenes, in-the-wild
DROID (Distributed Robot Interaction Dataset, 2024) tackles the diversity: ~76k trajectories collected in 564 scenes and 86 tasks across 13 institutions, with a standardized setup (Franka + RealSense). It is in-the-wild — real offices, kitchens, and labs — not a controlled table.
📊 Why diversity matters
- 76k — trajectories on standardized hardware (Franka).
- 564 — distinct scenes; robust visual generalization.
- 86 — tasks; broad skill coverage.
- in-the-wild — Real-world environments reduce overfitting to a single table.
⚡ Practical tip
Standardized hardware (Franka + Robotiq + RealSense) is the secret to DROID: the action space is consistent across all 564 scenes, so all the variance goes into what matters—scenario and task, not robot calibration.
DROID
76k diverse demos.
564 scenes
Visual variety.
Standardized
Franka + RealSense.
In-the-wild
Real environments.
🔹 LIBERO: continuous learning benchmark
LIBERO is less a training dataset and more a transfer benchmark: measures how a policy learns new tasks without forgetting old ones (lifelong learning). The four suites isolate dimensions — Spatial, Object, Goal e Long (long horizon).
📊 What each suite isolates
- Spatial — same task, different spatial layout.
- Object — same layout, different objects.
- Goal — same scene, different instruction/goal.
- Long — long horizons, composing subtasks.
# suites do LIBERO (cada uma isola uma variação)
libero_spatial # mesma tarefa, layout espacial diferente
libero_object # mesmo layout, objetos diferentes
libero_goal # mesma cena, objetivo/instrução diferente
libero_long # tarefas de horizonte longo (10 subtarefas)
# métrica: success rate por suite -> robustez de transferência⚡ Practical tip
LIBERO is the de facto benchmark for reporting VLA fine-tuning. A high score only on spatial and low in long reveals that the policy does not compose skills—a sign of weakness over long horizons.
LIBERO
Lifelong learning.
4 suites
Spatial/Object/Goal/Long.
Transfer
No catastrophic forgetting.
Long horizon
Skill composition.
🔹 Format and normalization: RLDS/TFDS, heterogeneous action spaces
Combining 60 datasets requires a common format: RLDS (Reinforcement Learning Datasets) on TFDS. The challenge is harmonization: each robot has its own action space (joints vs. Δpose, binary vs. continuous gripper, different frequencies). Without normalizing to a shared representation, co-training does not transfer.
import tensorflow_datasets as tfds
# carregar uma fonte do OXE (formato RLDS)
ds = tfds.load("fractal20220817_data", split="train") # RT-1 dataset
for ep in ds.take(1):
for step in ep["steps"]:
step["observation"]["image"] # frame
step["action"] # action space do robô-fonte
step["language_instruction"] # instrução textual
# harmonizar: normalizar ações p/ representação comum (ex: EEF Δpose 7D)📊 Heterogeneity dimensions
- Action space — joints vs. EEF Δpose vs. velocity.
- Gripper — binary, continuous, force.
- Frequency — 3 Hz to 50+ Hz across datasets.
- Cameras — number, resolution, viewpoint.
RLDS
Standard RL format.
TFDS
Dataset pipeline.
Harmonization
Shared action space.
Normalization
Comparable scales.
🔹 Curation and licenses: quality, bias, how to contribute
More data isn't better data. Curation — filtering failed demos and balancing coverage often works better than increasing raw volume. Pay attention to embodiment bias (datasets dominated by one robot) and sometimes licenses from each source. Contributing is simple: standardize in the format and publish on the Hub.
Filter for quality
Remove failed/noisy demos; suboptimal ones confuse behavior cloning.
Check bias/license
Balance embodiments; respect the terms of each source dataset.
Contribute
Standardize (LeRobotDataset/RLDS) and publish to the Hub with a dataset card.
⚡ Practical tip
Your SO-101 demos (M2.2) can join this ecosystem. Standardized and published, they become fuel for the next VLA — and close the loop between tools and data.
Curation
Quality > volume.
Bias
Unbalanced embodiment.
License
Terms by source.
Contribute
Standardize + publish.
✓ Do
- ✓Filter failed demos before training.
- ✓Balance embodiments to reduce bias.
- ✓Publish with a dataset card and clear license.
✗ Avoid
- ✗Assuming more raw volume = a better model.
- ✗Mix suboptimal demos without labeling quality.
- ✗Ignore license terms of aggregated sources.
✅ Module summary
Next track
Track 3 — Advanced and Strategic: sim-to-real, humanoids, hybrid architectures, and the $111B market. You already have the tools; now explore the frontier.