🗺️ Model Overview: RT-2, OpenVLA, π0, Qwen-VLA
The VLA model ecosystem mapped end to end—from Google’s closed pioneers to compact open-source models that run on a CPU. Understand what each model solves and when to choose each one.
Illustrative diagram — relative positioning of the main VLA models by openness and size.
🔬 RT-2 and RT-X: the pioneers at Google DeepMind
RT-2 redefined what a language model can do: instead of only describing the world, it age it. By co-training robotic data with web data at massive scale, Google DeepMind demonstrated that internet knowledge transfers to motor control — a result that surprised much of the community in 2023.
🔑 What RT-2 and RT-X are
- RT-2Robotics Transformer 2. First VLA to co-train robotic data with large-scale web data, demonstrating that internet knowledge transfers to motor control. Represents actions as text tokens, reusing the LLM's vocabulary.
- RT-XOpen X-Embodiment extension. Extends RT-2 by training on the Open X-Embodiment dataset with trajectories from 22 different embodiments, creating a cross-embodiment generalist model.
✓ RT-2's strengths
- ✓ Web-scale knowledge transfer: recognizes objects never seen in demos
- ✓ Emergent generalization: performs tasks without task-specific training
- ✓ Co-fine-tuning: web + robotic data in the same training loop
✗ RT-2 limitations
- ✗ Closed model: weights and code are not publicly available
- ✗ 55B parameters: prohibitive inference infrastructure
- ✗ High latency: insufficient control frequency for high-dexterity tasks
💡 Why study RT-2 even though it's closed
RT-2 defined the paradigm that virtually all VLAs follow today. Understanding its design choices and limitations provides context for why the community sought open and efficient alternatives like OpenVLA, Octo, and SmolVLA. It is an essential historical reference point.
🔓 OpenVLA: Stanford's open-source standard
If RT-2 showed what was possible, OpenVLA showed that it was accessible. A fully open 7B-parameter model, trained on 970 thousand trajectories from Open X-Embodiment, with performance competitive with RT-2-X using a fraction of the resources—and with weights, code, and fine-tuning recipe available to anyone.
🏗️ Three-layer architecture
SigLIP + DINOv2 combined: semantic + geometric features for fine manipulation.
7B-parameter backbone: reasoning and natural language instruction understanding.
Actions mapped to 256 bins per dimension, emitted as tokens from the LLM’s vocabulary.
💡 Fine-tuning on a consumer GPU
Efficient fine-tuning via LoRA runs on a single consumer GPU (e.g., RTX 4090). INT4 quantization enables inference at ~6 Hz on modest hardware. OpenVLA is the starting point for any real project with a limited budget.
📊 Key result from the original paper
Co-fine-tuning (vision + language + action together) improves performance by up to +16,5% compared with fine-tuning only the action head, justifying the extra computational cost. The dual encoder outperforms single encoders on fine manipulation tasks across multiple benchmarks.
🌊 π0 (pi0): flow matching from Physical Intelligence
π0 answers the question: what if the action head weren't discrete, but continuous and generated by a denoising process? By using flow matching instead of action tokenization, Physical Intelligence achieved 50 Hz control with smooth trajectories—necessary for high-dexterity tasks such as folding clothes.
π0 innovation timeline
VLM
PaliGemma as backbone
π0 uses PaliGemma (Google's VLM) for perception and reasoning, drawing on high-quality web data for pretraining.
Expert
Coupled action experts
Dedicated action-generation subnetworks receive tokens from the VLM backbone and produce continuous trajectories via flow matching — without discretization.
Match
Generation at 50 Hz
The vector field learns to transport Gaussian noise into actions via an ODE. The result: control at 50 Hz with smoothness that discrete tokenization cannot match.
🔑 Key π0 concepts
Learns a vector field that transports Gaussian noise to the action distribution via an ODE. More stable than diffusion and faster at inference.
Enables high-dexterity tasks (folding clothes, assembling boxes) with near-continuous feedback — impossible with discrete tokenization.
Designed for multiple robots: a single checkpoint works across different morphologies without retraining everything from scratch.
Excels at fine manipulation tasks where discrete VLAs fail: fluid, multimodal movements in continuous action space.
Illustrative diagram — discrete action head in steps vs. continuous action head with flow matching.
🐙 Octo: Berkeley's generalist transformer
While RT-2 and π0 aim for maximum capability, Octo takes the opposite approach: what is the smallest model capable of generalizing? With 27M to 93M parameters—hundreds of times smaller than RT-2—it showed that modular architecture and quality data matter more than raw scale.
🧩 Modular architecture
New sensors and action heads can be added without retraining the core. A transformer with attention blocks over observation and task tokens — pluggable and extensible.
🎯 Diffusion action head
Generates smooth multimodal actions through an iterative denoising process. Supports conditioning by language or by goal image—more flexible than discrete binning.
📦 800k Open X-E trajectories
Trained on 800 thousand trajectories from Open X-Embodiment, covering multiple robots and environments and enabling zero-shot transfer to new embodiments.
💻 Deploying on modest hardware
93M parameters run on accessible GPUs. Ideal for rapid research and prototyping when OpenVLA is too large for the compute budget.
💡 When to use Octo
Use Octo when you need a quick-to-train and test baseline or when your GPU budget is limited. It proves that not every VLA needs billions of parameters — the right architecture + the right data > raw scale.
📱 Qwen-VLA, SmolVLA, and the era of compact models
The third generation of VLAs isn't trying to beat benchmarks on data center servers—it's trying to run on low-cost embedded hardware. SmolVLA runs on CPU; Qwen-VLA balances capability and latency at 1–3B parameters. The question is no longer “how good?” but "how good is it with the hardware I have?"
✓ What compact models solve
- ✓ On-device inference without a cloud GPU
- ✓ Fine-tuning on a single consumer GPU
- ✓ Prototyping with low-cost platforms (SO-100, Jetson)
- ✓ Knowledge distillation preserves essential capabilities
✗ What compact models don't deliver
- ✗ Competitive zero-shot generalization with large models
- ✗ High dexterity in complex fine manipulation tasks
- ✗ Robustness to scenarios far outside the training distribution
🔑 Techniques that make compact models viable
- KDKnowledge distillation: transfers capabilities from large models to small ones, preserving reasoning without parameter cost.
- INT4Aggressive quantization: 4-bit weights reduce memory footprint by ~8x with minimal loss of performance on structured tasks.
- AsyncAsync inference (SmolVLA): perception and action generation run in separate threads — improved throughput without increasing the number of parameters.
⚖️ Comparison: Size, License, Embodiments, Performance
There is no single best VLA model — there is the best model for each constraint. The wrong choice costs weeks of work and thousands of dollars in compute. Understanding the trade-offs before you start is the difference between a project that moves forward and one that gets stuck.
📊 How models are evaluated
- LIBEROContinual learning benchmark for manipulation; measures retention of previous tasks while learning new ones. Standard for comparing efficient fine-tuning.
- SimplerEnvSimplified environment with standardized success rate metrics; enables fair comparison between models in controlled lab scenarios.
- Embodiment coverageDetermines whether the model transfers to your specific robot without extensive fine-tuning — crucial for real projects with proprietary hardware.
✓ Criteria for making the right choice
- ✓ Define compute constraints first
- ✓ Check embodiment coverage
- ✓ Check whether the license allows commercial use
- ✓ Check control frequency against task requirements
✗ Common selection mistakes
- ✗ Choosing the largest available model without checking the budget
- ✗ Ignore inference latency versus the required frequency
- ✗ Using a closed model in a project that requires reproducibility
- ✗ Overlooking embodiment compatibility
✅ Module Summary
Next module
1.4 — Datasets: Open X-Embodiment and DROID