🌫️ Diffusion Policy and Flow Matching

In Module 1.4, we saw that MSE collapses modes. Diffusion solves this by modeling the action as a sampling from a conditional distribution: instead of predicting a value, it learns to transform Gaussian noise into a valid trajectory through denoising. As a sample, it naturally captures multimodal distributions — chooses one coherent mode instead of the average. And by generating the entire trajectory (chunk), it produces smooth, consistent movements.

Diffusion Policy (Chi et al., 2023) applied DDPM to control: a network (1D U-Net or transformer) denoises a sequence of future actions conditioned on the observation. Combines action chunking (predict H steps) with receding horizon (execute part, re-predict). It beat baselines in fine manipulation and became the gold standard for dexterous tasks—including Octo's action head.

# Diffusion Policy — inferência (esboço)
A = torch.randn(H, action_dim)          # ruído inicial
for k in reversed(range(K)):            # K passos de denoising
    eps = net(A, k, obs_cond)           # prevê o ruído
    A = ddpm_step(A, eps, k)            # remove um pouco
execute(A[:exec_horizon])               # receding horizon

Under the hood, diffusion has a forward process that adds Gaussian noise to the action over K steps until it becomes pure noise, and a reverse process learned to undo it. The network learns the score — the log-density gradient ∇log p(a) — equivalent to predicting the added noise. The conditioning the observation enters via FiLM or cross-attention, making generation depend on what the robot sees.

Flow Matching generalizes diffusion: instead of a stochastic denoising process, it learns a vector field that transports noise→action through a Continuous ODE. With conditional flow matching, the training target is simply the velocity along a path (often straight) between noise and data—a simpler, more stable objective. The π0 uses exactly this in its action expert to generate smooth continuous actions at 50Hz.

The central cost of generative policies is number of function evaluations (NFE) — Each denoising/integration step is a forward pass. Classic DDPM uses tens to hundreds of steps, impractical at 50Hz. Solutions: fast samplers (DDIM), consistency models (generate in 1-2 steps) and distillation from a slow model to a fast one. Flow matching with straight paths already reduces NFE naturally.

The key design choice for the action head: diffusion/flow generation versus autoregressive tokens. Diffusion is more expressive (continuous multimodal output) but requires multiple steps. Autoregressive methods (RT-2, OpenVLA, FAST) reuse the LLM backbone and are simple to train, but generate one token at a time, and discretization can limit fidelity. There is no universal winner—the trade-off is expressiveness × latency × ease of integration.

Eixo            Difusão/Flow        Autoregressivo (tokens)
expressividade  alta (multimodal)   média (depende da disc.)
latência        K passos por chunk  1 forward, N tokens
integração      action expert extra reusa o backbone LLM
exemplos        Diffusion Policy,π0 RT-2, OpenVLA, FAST