🌍 From LLMs to Robots That Act: The Big Picture
The evolution of AI: from models that only write text to agents that see the world and manipulate physical objects. This module builds the mental map that underpins the rest of the course.
Illustrative diagram — each step adds a modality: first vision at the input, then action at the output.
🧬 From language to action: LLM → VLM → VLA
The progression from models that process only text (LLMs) for models that also see (VLMs) and, finally, for models that generate motor actions in the real world (VLAs). Each leap adds a modality — understanding this chain is the foundation for everything that comes later in the course.
🔑 The three acronyms you’ll never forget
- LLMLarge Language Model. Processes sequences of text tokens. Text goes in, text comes out. E.g.: GPT-3, LLaMA.
- VLMVision-Language Model. Add a visual encoder to process images alongside text. E.g.: GPT-4V, LLaVA.
- VLAVision-Language-Action. Extends the VLM with a action head that decodes tokens into continuous motor commands—position (x,y,z) and gripper rotation.
💡 Why this matters
Without clarity on what changes between GPT-4V and RT-2, it's impossible to evaluate VLA architectures or decide which model to use in a real project. The difference isn't "one more feature"—it's a new output modality: action.
🪢 The gap between understanding and acting
There is a chasm between semantic understanding (knowing what a cup is) and motor skill (pick up the cup without dropping it). VLMs understand scenes precisely, but they don't translate that understanding into coordinated physical actions. VLAs exist to close this gap.
✓ What a VLM already handles
- ✓Recognizing that there is a cup on the table
- ✓Describe the scene in natural language
- ✓Answer "where is the cup?"
✗ What’s still missing (and what VLA tackles)
- ✗Calculate the arm's trajectory to the cup
- ✗Adjust the gripper force in real time
- ✗Correct the motion when the cup slips
Three concepts for naming the gap
Ground abstract concepts in concrete sensory perceptions.
Difference between performance in simulation and in the real world.
Continuously adjust actions based on sensory feedback.
🏗️ Foundation models for robotics
The paradigm of foundation models (large-scale pretraining + fine-tuning) applied to robotics. Instead of training from scratch for each task and robot, you start with a base model trained on massive datasets from multiple robots and environments — and then adapt it. That's why VLAs have become viable.
📊 Why pretraining transfers so well
- Transfer learning — reuses knowledge from data-rich domains (the internet) for data-scarce domains (robotics).
- Multimodal pretraining — combines vision and language into a unified representation.
- Cross-embodiment — a single model controls robots with different morphologies.
💡 Practical tip
Rich representations from the internet transfer surprisingly well to motor control — even with few demonstration data points. In practice: you rarely train a VLA from scratch; you fine-tune a base model.
🖐️ Affordances: what the robot CAN do
A concept from J.J. Gibson's ecological psychology adapted for robotics: affordances are the action possibilities that an object or environment offers. A cup "affords" being grasped; a door, being opened. Detecting affordances means identifying which actions are physically feasible in the current scene.
Illustrative diagram — affordance detection distinguishes what is possible from what is merely requested.
Key concepts
- Visual affordances — map regions of the image to possible actions (graspable surfaces, navigable areas).
- SayCan (Google) — combines an LLM that proposes actions with a value function that filters the physically feasible ones.
- TAMP — Task and Motion Planning integrates symbolic planning with continuous motion planning.
🤖 Embodied AI vs. Disembodied AI
The distinction between AI that exists only in the digital world (chatbots, image generators) and AI embodied in physical bodies that interact with the real world. Working with VLAs requires a change in mindset: an error isn't a wrong answer in a chat — it's a collision, a dropped object, a safety risk.
Embodied AI
- •Error = wrong answer, reversible
- •No physics, no sensory latency
- •Can "try again" at no cost
Embodied AI
- •Error = collision, fall, damage—irreversible
- •Newtonian physics, mechanical wear
- •Real consequences for every action
Key concepts
True intelligence requires a body that interacts with the environment.
Patterns between actions and sensory changes that the agent learns.
The gap between simulation and the real world makes deployment difficult.
📅 Timeline: from GPT to autonomous humanoids
In just 3 years we went from "LLMs don't control robots" to "general-purpose VLAs control multiple embodiments." Knowing the timeline helps calibrate expectations and project where we'll be in 2027–2028.
GPT-3 and the LLM era
Large language models exhibit emergent capabilities. The idea of a "foundation model" is born.
RT-2 (Google DeepMind)
First prominent VLA: transfers web knowledge to real robotic control. The action becomes "just another token."
OpenVLA, π0, and Physical Intelligence
Open-source democratizes access. Physical Intelligence (π0) emerges. LeRobot and Octo open up the ecosystem.
Autonomous humanoids
Figure 02, GR00T, and the explosion of robotics startups. General-purpose VLAs control multiple embodiments.
💡 Concepts to follow along the learning curve
Scaling laws investigate whether more data/parameters improve motor performance as they do language. Emergent capabilities are abilities that emerge with scale without being explicitly trained. The ecosystem open-source democratizes what once required labs with millions invested in hardware.
✅ Module Summary
Next module
1.2 — Anatomy of a VLA: Vision + Language + Action