Lesson 7 · 14 min
So far the path has gone from the simulator to the robot. The reverse path exists too: photograph the real place and build a twin where the robot trains and is tested first. In this lesson you run the twin from the robotica-real2sim project (an arm, not a humanoid) in the course venv and prove a plan in the physics, and you see where this stands for humanoids in 2026.
Start the lesson
Goal: run the robotica-real2sim twin in the course venv and prove a plan in the physics before the robot, counting the objects in the scene.
Click each block. This is the robotica-real2sim flow.
Real-to-sim is the subject of robotica-real2sim. Here it runs and is tied to the ladder: a twin (digital twin) of the place is one more step before the floor.
Status as of Oct 2026, according to the program's research. Click each item.
Niantic Spatial, Flexion and NVIDIA showed a scene scanned by camera turning into Gaussian splatting in Isaac Sim, with a humanoid navigation policy trained there and taken to the robot with no tuning. This is vendor material, not a paper with measurements.
Berkeley, ETH and Amazon FAR use 3DGS as a renderer inside a vectorized simulator, at 100k+ steps per second (arXiv 2510.15352). Also with no measured sim-to-real on a humanoid as of Oct 2026, according to the research.
What already makes it into day-to-day work is simpler: measuring the floor and the friction, weighing the robot, scanning steps and obstacles for the terrain curriculum (course 1).
A twin (digital twin) is only as good as the measurement that feeds it. The real2sim project measured this: different vision models got the tray size wrong, and the proof only approves what passes in the worst case.
Adds the Panda to the bench's Menagerie sparse-checkout, pins robotica-real2sim to the tested commit and points its vendor/menagerie to yours.
F=${FTH:-$HOME/fth}; cd $F && . .venv/bin/activate && pip install -q pillow imageio
git -C mujoco_menagerie sparse-checkout add franka_emika_panda
[ -d robotica-real2sim ] || git clone -q https://github.com/inematds/robotica-real2sim
git -C robotica-real2sim checkout -q 3a08537 && mkdir -p robotica-real2sim/vendor
ln -sfn $F/mujoco_menagerie robotica-real2sim/vendor/menagerie
cd robotica-real2sim && python3 r2s.py ver exemplos/mesa-foto/cena.json $F/gemeo.png --camera cima
python3 $F/sim2real/kit/gemeo.py exemplos/mesa-foto/cena.json $F/gemeo.pngF=${FTH:-$HOME/fth}; cd $F/robotica-real2sim && . $F/.venv/bin/activate
python3 r2s.py provar exemplos/mesa-foto/cena.json resultados/mesa-foto/plano.json -n 20 | tail -3imagem: ~/fth/gemeo.png cena: cena.json · mesa 1.2 × 0.8 m · objetos na cena: 4 cubo_vermelho caixa lata_azul cilindro bola_tenis esfera bandeja_cinza recipiente gemeo.png: 1280×720 px, 135 KB
Four objects, a 1280×720 image generated by the GPU (EGL), no screen.
plano completo em 19/20 (95%); mínimo 90% -> APROVADO para o robô real
lata_azul -> bandeja_cinza: 20 ok
bola_tenis -> bandeja_cinza: 19 ok {'objeto caiu fora do destino': 1}19 out of 20 (95%): approved. The failure is noted by cause, as in the gate in lesson 6.
^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'glGetError'
r2s.py renders on the GPU (MUJOCO_GL=egl). Without a GPU, switch to osmesa, but the OSMesa library must be installed; without it, you get the error above. Counting objects doesn't depend on rendering: gemeo.py reads cena.json.
A 10 s clip from robotica-real2sim: one episode with three cameras.
The real world inside the simulator.
Run the twin block and open gemeo.png.
Done when: python3 ~/fth/sim2real/kit/gemeo.py exemplos/mesa-foto/cena.json ~/fth/gemeo.png (in the robotica-real2sim folder) → objetos na cena: 4 and gemeo.png: 1280×720 px.
Prove the example plan in 20 variations.
Done when: python3 r2s.py provar … -n 20 | tail -3 → plano completo em 19/20 (95%); mínimo 90% -> APROVADO para o robô real.
For a G1 that will walk down a factory corridor, list three measurements of the real place that you would bring into the simulator before the ladder in lesson 8.
Done when: three items with what to measure and with which instrument (e.g., floor friction with an inclined plane; step height with a tape measure).
The explanation appears after your answer.
Question 1 of 3
What does real-to-sim do that sim-to-real does not?skill
Question 2 of 3
As of Oct 2026, real-to-sim with Gaussian splatting for humanoids is:
Question 3 of 3
Without a GPU, r2s.py ver gives an OpenGL error. How do you go on?
Pick one answer. The explanation appears after you try.