A photo becomes a simulation twin, the twin generates labeled training data, and the plan only goes to the real robot after it passes the physics.
robotica-real2sim is an open project that reproduces on your computer the idea Google showed in the robotics part of Gemini 4 Argon: take a photo of a table and turn it into a physical simulation where a robotic arm trains. It is for anyone who wants to understand and test robotics with AI without owning a robot. It builds the "twin" of the table, generates dozens of labeled attempts for training, and tests each plan in the simulation before sending it to a real robot. It needs Python, MuJoCo (a free simulator) and, to read the photo, Codex.
The bottleneck in robotics has always been data: people in VR headsets guiding the arm for hours. Here the data comes from simulation, and the simulation comes from a photo.
A vision model reads the photo, measures the objects using an A4 sheet as a ruler and builds the scene in MuJoCo with the Franka Panda arm.
A closed-loop motion generator picks up and drops each object with variations; every attempt comes out with 4-level annotation and a quality stamp.
The plan written by the AI runs 20 times in the physics. Below 90% success, it is rejected and never reaches the robot.
Each step is a r2s.py command and writes a file you can open and check.
Every 20 ms the generator reads where the object is and recomputes the target. If the object moves, the arm corrects. This is what Google calls a closed-loop IK trajectory generator.
Goal → subtasks (pick, carry, drop) → primitives with start and end → events (both fingers touching, object lifted, success).
Each episode samples position (±4 cm), rotation, mass (±30%). A model trained on this learns to handle a world that is not identical to the photo.
Published on 13/10/2026. The post is a collection of examples with video; full sources in docs/PESQUISA.md (in Portuguese).
gemini-4-argon, US$ 2 / US$ 10 per million tokens at launch pricing) and in Antigravity.| Example | What happens | Number |
|---|---|---|
| One photo → better VLA | Photo of a workbench with 2 Franka FR3 arms becomes a MuJoCo twin, 77 tasks, closed-loop IK generator, 4,500+ episodes to fine-tune a VLA | 56% → 75% success on the real robot; 35 s → 24 s |
| ERQA | Embodied reasoning benchmark | 87%, above the other frontier models |
| Spot | Creates the twin of a pedal bin, trains 3-legged balance in MuJoCo and only then steps on the real pedal | no published rate |
| SO-101 | Cheap arm improves its own task by looking at logs and video, under the rules of an AGENTS.md | 58.9 s → 22.9 s → 17.4 s |
| ALOHA 2 | 136 s origami annotated on 4 levels with quality control | 1 goal, 6 subtasks, 17 primitives |
| DroneBench (Andon Labs) | Models write the code for a US$ 129 drone to find and follow a person | best model at 84% of the way; nobody completes all 5 tasks |
Runs on Linux, macOS and WSL. Tested on a 20-core ARM with GPU (rendering via EGL).
The installer creates the environment and downloads the Panda from MuJoCo Menagerie.
# check python3 --version && git --version
On a server or WSL, MuJoCo draws through the GPU. Without a GPU, use osmesa.
# project default export MUJOCO_GL=egl
Only to read the photo and write the plan. Without it, write the scene by hand.
# once codex login
Real commands. Each one prints what it did and saves the result.
Creates the .venv, installs MuJoCo, downloads the Franka Panda and runs the 8 tests (~11 s).
git clone https://github.com/inematds/robotica-real2sim && cd robotica-real2sim ./instalar.sh # ends with "TUDO OK"
Photograph the table from above, with an A4 sheet visible as a ruler. The model returns each object with type, size and position; the script converts to the arm's reference frame and warns if it had to move something closer to within reach.
./r2s.py foto minha-mesa.jpg cena.json --pedido "pôr o cubo vermelho na bandeja" # cena: 4 objetos -> cena.json ./r2s.py ver cena.json gemeo.png --camera cima # compare with the photo
Films an episode with three cameras (overview, top-down and wrist) and the phase written on screen.
./r2s.py filmar cena.json episodio.mp4 --cameras geral,cima,pulso # sucesso: objeto dentro do destino (269 quadros)
N episodes with variation. --imagens also saves the top-down and wrist cameras (128×128, 10 Hz). A failed episode is left with peso_no_treino: 0.
./r2s.py episodios cena.json dados/ -n 50 --imagens # 50/50 episódios com sucesso (100%) em 8.7 s
Codex reads the scene and returns pick → destination steps. The script rejects a plan that cites a nonexistent object or a destination that is not a container.
./r2s.py planejar cena.json "arrume a mesa: tudo que é redondo vai para a bandeja" plano.json # pegar lata_azul -> bandeja_cinza # pegar bola_tenis -> bandeja_cinza
The plan runs 20 times with variation. Without slots, the ball lands on top of the can. With one slot per object (the default), it passes.
./r2s.py provar cena.json plano.json --sem-vagas # plano completo em 4/20 (20%) -> REPROVADO: não mande ao robô ./r2s.py provar cena.json plano.json # plano completo em 19/20 (95%) -> APROVADO para o robô real
Files in resultados/ (in Portuguese). The example photo was generated by AI.
| Measure | Result |
|---|---|
| Episodes on the photo scene (cube → tray, with variation) | 50/50 in 8.7 s |
| Episodes on the test scene | 50/50 in 8.4 s |
| Each object alone (cube, can, ball × 2 scenes, 20 variations) | 20/20 in all |
| Plan "everything round goes to the tray", without slots | 4/20 (20%) → rejected |
| Same plan with one slot per object | 19/20 (95%) → approved |
Until the can worked, the simulation went wrong in three ways: a grasp that was too high pushed the can into the palm, the cylinder × finger contact pushed in the wrong direction and the model's gripper squeezed only ~3 N (the real one does 70 N). Each error would give a false number. That is why a twin has to be checked.
Codex, Claude, Gemini Robotics-ER 2 and Gemini 4 Argon built the scene from the same photo and wrote the plan for the same request. Physics decided. Full data in docs/CODEX-VS-CLAUDE.md (in Portuguese).
| Model | Access | Photo → scene: input / output | Plan: input / output | Cost | Cube–tray | Proof without / with slots |
|---|---|---|---|---|---|---|
| Codex gpt-6-luna | codex exec | 26,760 / 279 | 24,741 / 100 | subscription | 78 cm | 4/20 / 19/20 |
| Claude claude-opus-5-5 | claude -p | 174,040 / 3,928 111,596 cache | 116,476 / 662 20,936 cache | subscription | 48 cm | 16/20 / 16/20 |
| Gemini Robotics-ER 2 gemini-robotics-er-2-preview | Gemini API | 1,600 / 433 +1,538 reasoning | 377 / 70 +1,145 reasoning | US$ 0 · free | 45 cm | 0/20 / 16/20 |
| Gemini 4 Argon google/gemini-4-argon | OpenRouter | 1,614 / 542 | 389 / 72 | US$ 0.0101 | 41 cm | 7/20 / 20/20 |
Full version, with sources and what we measured, in docs/QUESTOES.md (in Portuguese).
Jack Clark (Anthropic) saw a 60% chance of recursive self-improvement by 2028; an aggregator says he moved it up to 2027, with no original source. What Google shows is task improvement, not weights. The useful question: who approves the next round?
A US$ 129 drone and AI-written code already follow a person better than the human reference code. Andon Labs itself says the public must decide the limits.
It can: RoboPAIR jailbroke three robots, with success "often 100%". The defense has to sit outside the model: force limit, forbidden zone, physical stop and proof in simulation.
Google shows a single real number (56% → 75%). Here, three physics errors would have given false results. Always ask which real-robot number supports the result.
If data gets cheap (50 episodes in 9 s), power shifts to whoever has the model that generates the data and the hardware that validates it. The counterweight is open: MuJoCo, LeRobot, SO-101, OpenVLA.
The first job hit is the teleoperator who generates data and the engineer who writes simulation by hand. The new role is reviewing and calibrating what the model produces.
In Google's example, Argon builds the environment and checks reward == 1.0 in it. DroneBench records cheating in up to 66% of a model's runs. Separate who builds, who judges and who checks.
In the EU, the Machinery Regulation applies from 20/01/2027 and requires third-party assessment when safety depends on AI. The practical minimum: record the model, the simulation that approved it and the real-world result.
Yes, and the gains are already significant. In simulation, multiple copies of the humanoid train in parallel without wearing out the equipment. Full text, with sources and review, at docs.HUMANOIDES.md.
With synthetic + real data, compared with real data alone.
Artificial movements created in 11 hours.
In terms of the duration of human demonstrations.
6,500 h of trajectories in 11 h. This is data generation, not the entire training process running faster.
NVIDIA data, March 2025 (technical blog).
| Capability | Potential gain |
|---|---|
| Learning to walk | Experiment with falls and terrain without breaking the robot |
| Balance | Simulate pushes, stumbles, and different loads |
| Manipulation | Learn to pick up objects of various sizes |
| Hand coordination | Generate many attempts at fine movements |
| New environments | Prepare behaviors before deployment |
| Safety | Test dangerous situations virtually |
On an RTX 4090, each second of computing is worth about 27 minutes of virtual experience, with 4,096 environments in parallel. This comes from parallelism: the robot did not become 1,620× smarter. This figure is based on the benchmark table and varies by version.
Including pushing and pulling a wheelbarrow. Each action was first simulated in two physics engines, and only approved actions were tried on the real robot. Locomotion, still without autonomy on the job site (arXiv 2608.01600).
The opportunity is not to compete with Isaac Lab: it is to automate the creation of the environment where the humanoid will learn. Film a kitchen or factory, generate the digital twin, prepare the tasks, train the virtual humanoid, validate it, and only then move to the robot. Reconstructing the place is not enough: contacts, doors and drawers, sensors, and the humanoid's own dynamics must be modeled. The closest reference is Video2Sim2Real (June 2026): a manipulation skill learned from a single human video, refined in simulation, and transferred to the robot.
How much does it cost, and how long does it take, to teach a humanoid a new task with Real2Sim2Real compared with teaching it through human demonstrations? With the same task and the same robot, this comparison proves much more than isolated simulation speed figures.
The photo sets up the workspace; the simulation does the teaching, with thousands of variations of the same task. Full text, with step-by-step instructions for the SO-101, at docs/APRENDER.md.
| Input | What it’s for |
|---|---|
| 1 photo | A simple table, everything in view. This is what already works here. |
| 3–4 photos from different angles | Better measurements and hidden objects. Measurements are where it makes the most mistakes. |
| Sequence of photos | Multistep task: each photo is a state the robot needs to reach. |
| Video of a person performing the task | Teaches the how, not just the workspace: human movement becomes robot movement, refined in the simulation. |
The AI writes "pick up the cube, take it to the tray, release it." Once per task.
Gripper position becomes the angle of each joint (inverse kinematics), 10 to 50 times per second.
Target position for each servo goes over USB to the arm board, hundreds of times per second.
Today, the programmed mode works: the AI plans, the simulation proves it, and the arm executes. Next is the learned mode: a neural network looks at the camera and joints and decides the next action in a continuous loop.
Each attempt is an episode: at each moment, what the robot saw and what it did.
episode 0007 · "Put the red cube on the gray tray."
t=0.0 cameras: top, wrist joints: [0.10 -0.50 0.80 …]
action: target joints [0.12 -0.48 0.79 …] gripper: open
t=0.1 …
end success = yes
Generated in 30 s from the photo of the table.
At 10 Hz, with state, action, and two cameras.
Through the official LeRobot 0.6.1 API, on a CPU.
Opens back up in LeRobot, ready to train.
./instalar.sh --lerobot ./r2s.py episodios exemplos/mesa-foto/cena.json eps/ -n 50 --imagens ./r2s.py exportar eps/ dataset/
You move the leader arm by hand, the follower copies it, and everything is recorded. With 50 to 100 demonstrations, an ACT policy can already learn a simple task.
Thousands of synthetic episodes add variety; a few real demonstrations correct the physics the simulation got wrong.
The robot tries, you take over when it makes a mistake, and the correction becomes new data (DAgger).
Each real-world failure goes into the dataset and the policy is trained again. Trial and error directly in the real world is possible, but it is slow and wears out the equipment.
pip install "lerobot[feetech,core-scripts,training]", then lerobot-find-port, lerobot-setup-motors, lerobot-calibrate, and lerobot-teleoperate to see the follower copy the leader.lerobot-record, varying object positions; --dataset.push_to_hub=false keeps everything local.lerobot-train --policy.type=act and lerobot-rollout on the arm, slowly and with the stop button at hand.--strategy.type=dagger to correct; success rate over 20 attempts: synthetic only, real only, and both together.The simulation arm is still the Franka Panda: the synthetic data applies directly to the SO-101 once you swap the model for the one from MuJoCo Menagerie.
What already exists and the path for anyone who wants to take it to the real world.
r2s.py exportar: 50 episodes with cameras become a dataset that LeRobot opens and trains on (in practice).r2s.py provar before a motor moves.