🧬 VLA Anatomy: Vision + Language + Action

The image comes in as a grid of patches (e.g., 16×16 px), each projected into a patch embedding — the visual token. A ViT processes these tokens with self-attention. Common choices serve different purposes: SigLIP aligns vision and text via contrastive learning (good for semantics/language), while DINOv2 learns dense self-supervised spatial features (good for localization and geometry). OpenVLA uses an dual encoder (SigLIP + DINOv2) specifically to combine semantics and spatial information.

The heart of a VLA is a Pretrained LLM which works as a multimodal reasoning engine. OpenVLA uses a Prismatic VLM based on Llama 2-7B; π0 starts from PaliGemma (3B). The advantage is inheriting billions of tokens of web knowledge—grammar, world knowledge, object relationships—that provide semantic grounding for the instruction. The backbone receives vision + text (+ state) tokens and produces a latent state that conditions the action.

How do vision and text enter the same context? In early fusion (OpenVLA, π0), visual and text tokens are simply concatenated in the LLM input sequence, which mixes them via self-attention — simple and effective, but the number of visual tokens increases the cost. In cross-attention (Flamingo, some VLAs), the text queries the image through dedicated cross-attention layers, keeping the language backbone leaner. The central trade-off is token budget versus fine-grained grounding capability.

# Early fusion (concatenação)
seq = [img_tok_1 ... img_tok_256, txt_tok_1 ... txt_tok_K]
latent = LLM(seq)            # self-attention mistura tudo

# Cross-attention
latent = LLM(txt_tok)        # texto no fluxo principal
latent = CrossAttn(latent, img_tok)  # consulta a imagem

The action head converts the latent state into motor commands — typically Δpose of the end effector (relative translation and rotation) plus the state of the gripper (open/close). There are three families: (1) detokenizer, when actions were discretized into tokens (RT-1/RT-2/OpenVLA); (2) action expert via diffusion/flow, a separate module that generates continuous actions (π0); (3) direct regression of floats. The choice determines expressiveness and latency — the topic of Modules 1.5 and 1.6.

A camera isn't enough. The robot needs to know its own state — joint angles, velocities, end-effector force and gripper state. These values go into a state token additional. Equally important is the temporal history: seeing only the current frame creates ambiguity (is the arm moving up or down?). Stacking recent observations gives the model a sense of movement and ongoing intent.

Fine manipulation requires commands at ~50Hz, but a 7B VLA doesn't infer as quickly. The solution is action chunking: the model predicts a block of H future actions at once (e.g., 16-50 steps), which are executed open-loop while the next inference runs. With receding horizon, re-predicts before the chunk ends, maintaining responsiveness. This amortizes network latency and smooths the trajectory—an idea introduced by ACT and adopted by Diffusion Policy and π0.