🔬 Anatomy of a VLA: Vision + Language + Action
Breaking down the architecture from the inside: how the visual encoder, language backbone, and action decoder connect to turn pixels and words into physical movement.
👁️ Visual Encoder: SigLIP, DINOv2, ViT
The visual encoder is the first step in perception: it converts each camera frame into dense vectors that the language backbone can "read." The choice of encoder—SigLIP, DINOv2, or pure ViT—determines what the robot can see accurately.
🔍 How patch embedding works
The 224×224-pixel image is divided into 14×14- or 16×16-pixel patches. Each patch is linearly projected into a dense, high-dimensional vector (typically 768 or 1024 dimensions). These vectors—the visual tokens — they're the internal language the encoder uses to communicate with the LLM backbone.
A 224×224 image with 14×14 patches produces 256 visual tokens. Each token carries local information about texture, color, and edges — together they carry the spatial structure of the entire scene.
✓ SigLIP and DINOv2 excel when
- ✓SigLIP: the task requires aligning objects with names ("pick up the blue cup")
- ✓DINOv2: the task requires depth estimation or implicit segmentation
- ✓Dual encoder (SigLIP + DINOv2): combines both — standard in OpenVLA
- ✓Higher resolution (336×336) improves fine manipulation of small objects
✗ Common pitfalls
- ✗Using an encoder trained only on web images — loses 3D geometry
- ✗Freeze the encoder during fine-tuning — features don’t adapt to the task
- ✗Very low resolution (112×112) — a 2 cm object becomes 1 token of noise
- ✗Ignore the projection layer — visual tokens need a scale compatible with the LLM
🧠 LLM Backbone: Llama, Gemma, Qwen
The LLM backbone is the VLA’s central brain. It fuses visual tokens with text tokens, reasons about the scene, and decides what to do. The choice of backbone — and its parameter count — defines the fundamental trade-off between reasoning ability and response latency.
💡 Why reuse a pretrained LLM?
Models such as Llama 3, Gemma 2, and Qwen 2.5 were trained on trillions of text tokens. They already "know" what a cup is, where the sink is, and how to follow compound instructions such as "put the fork to the left of the plate." Training this knowledge from scratch for each robot would be prohibitively expensive—reuse is the key to generalization.
Processing flow within the backbone
Projection Layer
A simple MLP (2 linear layers) maps visual tokens from the encoder space (e.g., 1024 dims) to the LLM embedding space (e.g., 4096 dims). This "translator" is the only new component that needs to be trained from scratch.
Multimodal Fusion
Visual tokens and text tokens are concatenated into a unified sequence. The transformer's attention mechanism treats both the same way — there's no architectural difference between "seeing" and "reading".
Representation Generation
The backbone processes the full sequence and produces rich output embeddings that encode: “the cup is 15 cm ahead, to the left of the plate, and the instruction says to pick it up.” These embeddings feed into the action head.
🎯 Action Head: Tokenizing Continuous Actions
The action head is where VLA diverges radically from a standard VLM. Robots operate in continuous spaces — position in meters, torque in Newton-meters — but LLMs generate discrete tokens. The action head is the component that resolves this fundamental incompatibility.
The continuity problem
A 7 DoF robotic manipulator needs 7 real numbers at every timestep: (x, y, z, rotX, rotY, rotZ, gripper). These numbers can be 0.3427m or 1.7293 Nm — arbitrarily precise values. An LLM’s vocabulary has discrete tokens like "cat," "42," or "[EOS]." There is no token for "0.3427."
Different action heads solve this in different ways: discretization into bins, direct regression via MLP, or generative models (diffusion, flow matching). Each approach has trade-offs in precision, latency, and smoothness of movement.
✓ Action heads that work well
- ✓Discrete bins (RT-2, OpenVLA): simple, compatible with LLM vocabulary, good for slow tasks
- ✓Diffusion head (Octo): smooth, multimodal actions, good when there are multiple valid paths
- ✓Flow matching (π0): high frequency (50 Hz), great for dexterous manipulation
✗ Common problems
- ✗Low-resolution bins (64 bins) — ~1.5 mm imprecision in a 10 cm workspace
- ✗Regression MLP without action normalization — gradients explode or collapse
- ✗Diffusion with many denoising steps — latency prohibitive for real-time control
💡 Flow matching vs. Diffusion
Both turn noise into actions. The difference: diffusion uses an iterative denoising process with many steps (typically 10-100); flow matching learns a vector field and solves an ODE in fewer steps (2-4). Result: π0 can generate smooth actions at 50 Hz, while typical diffusion heads need to trade off quality and speed.
📐 Trajectory Tokenization: Discrete Bin vs. Continuous
How can you represent a robotic trajectory as a processable sequence? Two fundamental approaches: discretize the action space into buckets (compatible with an LLM vocabulary, but limited in resolution) or generate continuous values directly via generative models (precise, but architecturally more complex).
📊 Real numbers: bin resolution
🔄 Co-fine-tuning: Vision + Language Together
Co-fine-tuning is the process of simultaneously training the visual and language components of a VLA on robotic demonstration data. Instead of freezing the encoder and training only the action head, all components learn together — and that is exactly what enables VLAs to outperform modular pipelines.
💡 OpenVLA’s result
OpenVLA empirically demonstrated that co-fine-tuning improves performance by up to 16,5% compared with fine-tuning only the action head. The reason: when the visual encoder can adapt to the task, it learns features specific to manipulation—such as detecting grasp edges—that web image features don’t capture.
✓ Benefits of co-fine-tuning
- ✓Gradients from the action loss propagate to the encoder—visual features adapt for control
- ✓Representations converge on what matters for the task, not web classification
- ✓Better generalization to new objects not seen in the demonstrations
- ✓LoRA makes co-fine-tuning feasible on a single consumer GPU
✗ Risks and mitigations
- ✗Catastrophic forgetting: backbone loses language capability → use a low learning rate (1e-5)
- ✗Overfitting: robotic data is scarce; with little data, the model memorizes instead of generalizing
- ✗Computational cost: co-fine-tuning a 7B model is 3-4× more expensive than fine-tuning only the action head
⚡ Latency and Throughput: Token Budget
The most punishing bottleneck for real-world VLAs isn’t the architecture—it’s time. A robot needs to decide on actions at 5–50 Hz, but a 7B-parameter LLM can take hundreds of milliseconds per token. The token budget determines how many visual and text tokens the system can process within the available control time.
Control frequencies and what each one enables
Slow tasks — stacking blocks, opening drawers
200 ms per action. 7B VLAs with INT8 quantization (e.g., OpenVLA) can operate at this frequency on consumer GPUs. Sufficient for 80% of tabletop manipulation tasks.
Fluid manipulation — picking up and placing varied objects
~67 ms per action. Requires smaller models (3B), aggressive quantization (INT4), or async inference with action chunking. KV cache is mandatory to reuse context tokens.
Dexterous manipulation — folding clothes, fitting parts together
20 ms per action. π0 territory with flow matching (2-4 ODE steps). Not feasible with bin discretization + a full-size LLM. Requires architecture and hardware co-design.
📊 Acceleration techniques and their gains
💡 Jetson Orin: VLA at the edge
The Nvidia Jetson AGX Orin (275 TOPS) can run SmolVLA or Qwen-VLA <3B with INT4 quantization at 8–12 Hz — enough for tabletop tasks — without any external server. For 7B models like OpenVLA, a discrete GPU (RTX 3090 or higher) is still needed in the loop.
🎓 Module Summary
Next Module:
1.3 — Model Landscape: RT-2, OpenVLA, π0, Qwen-VLA. The VLA model ecosystem, from closed pioneers to open source.