E1 · Run the script
5 minCopy orcamento.py, run it, and check that the output matches the one in the box.
Done when: your terminal shows 20 ms, 500 ms, 9, and "yes".
Lesson 2 · 16 min
Eye, brain, and hand: inside a VLA
Every VLA has three parts: one that sees, one that understands, and one that turns the decision into movement. The movement part is where models differ the most, and where the clock gets tight: the arm waits for a command every few milliseconds.
Start the lesson
In this lesson you will
- 1The three parts: visual encoder, backbone, and action head
- 2Three ways to generate action: bins, FAST, and flow matching
- 3Why the model returns a piece of actions (chunk)
- 4The slow/fast hierarchy: S2, S1, and S0
Goal: explain the path from image + sentence → actions and why action chunking and flow matching exist.
01The path from camera to motor
Scroll the page: each step lights up a part of the diagram.
Scroll the page: each step below lights up its part of the diagram.
- 1/5
Visual encoder
Turns each image into a sequence of vectors that the rest of the model understands. It usually comes ready-made from a pretrained VLM.
- 2/5
Backbone
A VLM that takes the image tokens, the instruction, and the robot's state. 2026 examples: GR00T N1.7 uses Cosmos-Reason2-2B; RDT2 uses Qwen2.5-VL-7B. This is where the "understands" in the course title comes from.
- 3/5
Action head
Turns the backbone's decision into motor numbers. You can pick values from a list (bins), write the compressed trajectory as tokens (FAST), or generate the continuous action with flow matching or diffusion.
- 4/5
Action chunk
Instead of one action at a time, the model returns a chunk of N actions. The controller executes that chunk while the model computes the next one.
- 5/5
The whole path
The camera becomes tokens, the backbone combines them with the sentence and the state, the action head returns a chunk of actions, and the motor executes while the next chunk is computed.
In most 2026 VLAs the backbone is a pretrained VLM, and what changes from one model to another is the action head.
02Three ways to generate the action
Click each one. The difference is speed and precision.
1. Bins
Each joint has its range of motion divided into little boxes (bins), and the model picks a box the way it would pick a word. Models like OpenVLA (7B, Jun/2024) follow this idea.
It's simple, but it gives one action at a time, and precision is limited by the size of the box.
2. FAST
Physical Intelligence's FAST tokenization applies a transform (DCT) and compression (BPE) to a piece of the trajectory, which becomes a few tokens. According to PI, it trains 5 times faster.
In May/2026, Ai2's MolmoAct 2 published an open reimplementation of the FAST tokenizer, with open data.
3. Flow matching
A part of the model dedicated to action (the action expert) starts from noise and pushes it toward the right action, in continuous values. This is the π0 approach.
Diffusion is a close relative: GR00T uses a 32-layer DiT (diffusion transformer), and the model from Boston Dynamics and TRI controls the entire Atlas at 30 Hz with a diffusion transformer and flow matching (Aug/2025).
Rule of thumb
Fine, fast movement: continuous action in chunks. Step-by-step reasoning: tokens.
4. Slow and fast
A big model thinks slowly; a motor wants fast commands. The solution is to stack systems. In Figure's Helix (Feb/2025), S2 is a VLM at 7–9 Hz and S1 is a policy at 200 Hz.
Helix 02 (Jan/2026) added S0: a 10M-parameter whole-body controller at 1 kHz, which replaces 109,504 lines of C++.
03The control time budget
A ten-line script that answers: is there time for the model to think before the chunk runs out?
orcamento.py
# orcamento.py: how much time each chunk of actions covers
hz = 50 # control frequency (commands per second)
chunk = 25 # actions the model returns at once
inferencia_ms = 180 # time for the model to generate a chunk
periodo_ms = 1000 / hz
dura_chunk_ms = chunk * periodo_ms
acoes_na_espera = inferencia_ms / periodo_ms
print(f"control period: {periodo_ms:.0f} ms")
print(f"a chunk of {chunk} actions covers: {dura_chunk_ms:.0f} ms")
print(f"actions that go by while the model thinks: {acoes_na_espera:.0f}")
print("is there time to think before the chunk runs out?", "yes" if inferencia_ms < dura_chunk_ms else "no")Terminal
python3 orcamento.py
Example from the repository
$ python3 orcamento.py control period: 20 ms a chunk of 25 actions covers: 500 ms actions that go by while the model thinks: 9 is there time to think before the chunk runs out? yes
The 180 ms is the time of one MolmoAct 2 action call as reported by Ai2 (May/2026). On your robot, measure yours.
- Forgetting the wait: while the model computes, the robot keeps moving. If the chunk runs out first, the arm stops or repeats the last action.
- Mixing up Hz and ms: 50 Hz is 50 commands per second, one every 20 ms.
04Practice: your time budget
Do the math before running the script, then check it.
E2 · Chunk too short
10 minChange chunk to 8 and keep 50 Hz and 180 ms. Before running, write down on paper: how much the chunk covers, how many actions go by during the wait, and whether there is enough time.
Done when: your paper and your terminal say the same thing: 160 ms, 9 actions, and "no" (the chunk ends before the next inference).
E3 · The smallest chunk that works
5 minStill at 50 Hz and 180 ms, find the smallest chunk that gives enough time.
Done when: you got to 10 actions (200 ms, greater than 180 ms) and you can explain why 9 is not enough.
05Test what you learned
The explanation appears after your answer.
Question 1 of 4
A controller at 50 Hz with a chunk of 25 actions. How long does the chunk cover?skill
Correct
50 Hz = one command every 20 ms. 25 × 20 ms = 500 ms. The common mistake is multiplying 25 by 50.Not this one
50 Hz = one command every 20 ms. 25 × 20 ms = 500 ms. The common mistake is multiplying 25 by 50.Review: The control time budgetQuestion 2 of 4
Why does action chunking exist?
Correct
The large model takes hundreds of milliseconds; the motor wants a command every few milliseconds. The chunk covers that gap.Not this one
The large model takes hundreds of milliseconds; the motor wants a command every few milliseconds. The chunk covers that gap.Review: The path from camera to motorQuestion 3 of 4
Which action head generates the action as a continuous value, starting from noise?
Correct
Flow matching (π0) and diffusion (GR00T's DiT) generate continuous values. Bins picks from a list; FAST writes the compressed trajectory as tokens.Not this one
Flow matching (π0) and diffusion (GR00T's DiT) generate continuous values. Bins picks from a list; FAST writes the compressed trajectory as tokens.Review: Three ways to generate the actionQuestion 4 of 4
In Helix 02, what does S0 do?
Correct
S2 understands slowly, S1 decides the joint targets at 200 Hz, and S0 controls the whole body at 1 kHz. Each layer at its own pace.Not this one
S2 understands slowly, S1 decides the joint targets at 200 Hz, and S0 controls the whole body at 1 kHz. Each layer at its own pace.Review: Three ways to generate the actionQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Pick an option. The explanation appears after your attempt.
06Summary
Eye, brain, and handVisual encoder, backbone (a VLM), and action head.
The hand is where it changesBins, FAST, or flow matching/diffusion: a list, compressed tokens, or continuous.
The clock rulesA chunk of N actions at 50 Hz covers N × 20 ms, and it has to last longer than the inference.