Detailed content
Each step solves the previous bottleneck: web knowledge, openness, continuous action, and increasingly large datasets.
🟢 RT-1 (2022): transformer + FiLM
RT-1 (Robotics Transformer 1) proved the basic formula: images pass through an EfficientNet conditioned by FiLM (instruction feature modulation), TokenLearner compresses tokens, and a transformer predicts actions discretized into 256 bins per dimension. Trained on ~130k demos across 700+ tasks with 13 robots over 17 months, it achieved ~97% success on seen tasks and modest generalization. It was proof of concept that tokenizing actions works at scale.
📊 RT-1 in numbers
- ~130k teleoperation demonstrations.
- 700+ distinct instructions, 13 robots.
- 256 bins per action dimension · 35M parameters.
FiLM
Modular vision through instruction.
TokenLearner
Visual token compression.
Action binning
256 bins per dimension.
Proof of concept
Tokenize action at scale.
🔵 RT-2 (2023): co-fine-tuned VLM
RT-2 started from web-scale VLMs (PaLI-X up to 55B, PaLM-E) and co-fine-tuned them with robot data, treating actions as text strings in the same vocabulary. The co-fine-tuning (mixing web and robot data) preserves internet knowledge, generating emergent semantic generalization: choosing "the object that doesn't belong" or "the flag of country X." RT-2 also demonstrated chain-of-thought robotic, reasoning in text before acting.
✓ RT-2 gains
- ✓Semantic generalization to novel objects/instructions.
- ✓Multi-step reasoning via chain-of-thought.
- ✓Reuses the scale of web pretraining.
✗ Limitations
- ✗Closed: weights and code not released.
- ✗Expensive inference (up to 55B), difficult in real time.
- ✗Discrete actions limit fine, dynamic control.
Action-as-text
Actions as tokenized strings.
Co-fine-tuning
Web + robot in the same training run.
Robotic CoT
Reason before acting.
Semantic emergence
Undemonstrated skills.
🟡 Octo & OpenVLA (2024): open-source
2024 democratized the paradigm. Octo (Berkeley) is a small transformer (27M-93M) with a diffusion head, trained on Open X-Embodiment and easy to fine-tune. OpenVLA-7B (Stanford/Berkeley) uses a Prismatic VLM (Llama 2-7B + SigLIP/DINOv2) trained on ~970k OXE trajectories, outperforming RT-2-X with 7× fewer parameters. The key point: both allow efficient fine-tuning via LoRA and quantization, running on a GPU.
# Fine-tunar OpenVLA com LoRA (esquema)
from transformers import AutoModelForVision2Seq
model = AutoModelForVision2Seq.from_pretrained(
"openvla/openvla-7b", load_in_4bit=True) # quantização
model = get_peft_model(model, LoraConfig(r=32)) # adapta poucos params
# treina em demos do seu robô → política especializada⚡ Practical tip
For most real-world projects, start with OpenVLA + LoRA on your own demos. It’s the shortest path from "I have an arm" to "I have a policy," without training a 7B model from scratch.
Prismatic VLM
OpenVLA base.
OXE
Open X-Embodiment training set.
LoRA
Fine-tuning with few parameters.
Octo
Lightweight transformer with diffusion.
🌊 π0 / π0.5 (Physical Intelligence)
π0 broke away from discrete tokenization. On top of a base PaliGemma, couples a action expert trained by flow matching that generates continuous actions in chunks at up to 50Hz — ideal for dexterous tasks like folding clothes. π0.5 advances open-world generalization by training on heterogeneous data (including web data) for long-horizon tasks in homes it has never seen. It is the open state of the art in generalist manipulation for 2025.
VLM base + action expert
PaliGemma reasons; a dedicated expert generates the continuous action.
Flow matching at 50Hz
Fluid generation of action chunks for dexterous tasks.
π0.5: open-world
Generalizes to novel homes and long-horizon tasks.
Flow matching
Continuous action via ODE.
Action expert
Dedicated generation module.
Open-world
Generalization to novel scenes.
Dexterity
Fine-grained tasks such as folding laundry.
🧠 Qwen-VLA and the 2025 family
2025 brought VLAs built on Increasingly larger and more capable VLMs — families based on Qwen2-VL and similar models. The bet: the better the base model's multimodal reasoning, the better the VLA performs on tasks that require understanding scenes, counting, and spatial relationships. The concept of embodied reasoning — the model verbalizes a plan and grounds it in action. Models like Gemini Robotics-ER and Qwen-VLA variants exemplify this convergence of strong reasoning and control.
📊 Trends in 2025-2026
- Larger bases — Reasoning VLMs (Qwen2-VL, Gemini) as the core.
- Embodied reasoning — plan in language before acting.
- Dual-system — separate slow reasoning from fast control (see Track 3).
💡 Why scale the base model
The intuition is that the bottleneck for complex tasks is no longer the action head, but the understanding the scene and the objective. Larger bases carry better priors—as long as there's enough action data to ground them.
Reasoning-VLA
Reasoning + action in the same model.
Scaling base
Larger VLMs as the core.
Embodied reasoning
Verbal plan grounded in action.
Gemini Robotics
Google’s frontier VLA.
📊 Comparative timeline
Summarizing the lineage across five dimensions — size, data, tokenization, license, and focus. The trend is clear: larger bases, increasingly continuous/expressive action, and a divide between closed frontier models (RT-2, Gemini) and practical open models (Octo, OpenVLA, π0).
Modelo Ano Params Ação Licença
RT-1 2022 35M bins (256) fechado
RT-2 2023 até 55B tokens/texto fechado
Octo 2024 27-93M difusão open
OpenVLA 2024 7B tokens (bins) open (Apache)
π0/π0.5 2024 ~3B flow matching open
Qwen-VLA 2025 XL contínua/flow varia⚡ Practical tip
When choosing, weigh license + inference cost + action type. Slow task and limited demos? OpenVLA. Dexterous, fluid manipulation? π0 with flow matching.
Benchmark
Compare in LIBERO/SimplerEnv.
License
Open vs. closed determines how it’s used.
Params vs data
Scale of the base and corpus.
Action type
Bins, diffusion, or flow.
✅ Module summary
Next module
1.4 — Imitation learning: how these models learn from demos and why BC breaks.