🧪 Simulate before going real

With the real arm, every attempt costs something: you set up the table, wait for the movement, wear out the servos, and risk knocking something over. In simulation, the computer calculates the physics and runs hundreds of attempts while you have a coffee. That’s why this course has a simple rule: no plan or policy goes to the real arm without first passing through simulation.

How to read this map: the funnel narrows from top to bottom. The wide section (simulation) filters out almost all planning and code errors. The middle section (slow real-world testing) catches what the simulation does not represent well. Only what passes both reaches normal use.

Why learn this

AI-powered robots make mistakes in unpredictable ways. Simulation is a safe place to discover those mistakes, and it is also a data factory that reduces how many demonstrations you need to record by hand.

Key concepts

O MuJoCo is a free, open physics simulator maintained by Google DeepMind, light enough to run on a laptop without a dedicated graphics card. Its official model collection, MuJoCo Menagerie, already includes the SO-101, ready to use, with the same 6 joints as the real arm. You install it with pip and control everything in Python.

How to read this map: the box in the middle is the joint list, the same on both sides. The same command ("shoulder at 0.6 rad") applies to the real arm and the twin. If the twin and the real arm end up in different places with the same command, you have just measured the sim2real gap from topic 5.

Goal: install MuJoCo in a virtual environment, download only the SO-101 from Menagerie, and open the twin in a 3D window.

python3 -m venv .venv-sim
.venv-sim/bin/pip install mujoco numpy

# downloads only the SO-101 folder from the official model collection
git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/google-deepmind/mujoco_menagerie.git
git -C mujoco_menagerie sparse-checkout set robotstudio_so101

# 3D window: drag to rotate, double-click a part, and Ctrl+drag to push it
.venv-sim/bin/python -m mujoco.viewer --mjcf=mujoco_menagerie/robotstudio_so101/scene.xml

Now control the same twin with code: move two joints to a position and measure where the gripper stopped.

# SO-101 twin in MuJoCo: load the model, move joints, and measure where the gripper ended up.
import sys
import mujoco

CENA = sys.argv[1] if len(sys.argv) > 1 else "mujoco_menagerie/robotstudio_so101/scene.xml"
model = mujoco.MjModel.from_xml_path(CENA)
data = mujoco.MjData(model)

print("juntas:", [model.joint(i).name for i in range(model.njnt)])
print("atuadores:", [model.actuator(i).name for i in range(model.nu)])

mujoco.mj_resetData(model, data)
alvo = {"shoulder_pan": 0.6, "elbow_flex": 0.5}   # radians (0.6 rad = about 34 degrees)
for nome, valor in alvo.items():
    data.ctrl[model.actuator(nome).id] = valor

garra = model.body("gripper").id
for passo in range(int(2.0 / model.opt.timestep)):  # 2 seconds of physics
    mujoco.mj_step(model, data)
    if passo % int(0.5 / model.opt.timestep) == 0:
        x, y, z = data.xpos[garra]
        pan = data.qpos[model.joint("shoulder_pan").qposadr[0]]
        print(f"t={data.time:4.1f}s  pan={pan:+.3f} rad  garra=({x:+.3f}, {y:+.3f}, {z:+.3f}) m")

erro = abs(data.qpos[model.joint("shoulder_pan").qposadr[0]] - alvo["shoulder_pan"])
print("final shoulder_pan error:", round(erro, 4), "rad", "OK" if erro < 0.05 else "FAR FROM TARGET")

Why learn this

With the twin, you can test reach, camera position, and project logic before printing a part. It also lets the whole class work with the arm when the school has only one.

Key concepts

O NVIDIA Isaac Sim is a simulator with near-photographic images, detailed sensors, and ROS 2 integration. The Isaac Lab, built on top of it, runs thousands of robot copies in parallel on the GPU, making reinforcement learning feasible (the robot learns through trial and error, guided by rewards). The trade-off is the hardware: it requires an NVIDIA RTX GPU with plenty of memory and a much heavier installation.

Why learn this

Choosing a simulator based on the problem, not its reputation, saves weeks of installation time. Most of what this course covers works in MuJoCo; use Isaac when the project calls for scale or realistic images.

Key concepts

Building a scene by hand (where the box is, how big the cube is) takes work. Real2sim is the reverse of sim2real: you start from the real world to build the simulation. A vision model looks at a photo of the table, estimates objects, sizes, and positions in meters, and a program builds the scene in MuJoCo. The simulation then generates episodes from it and tests the plan before the arm moves.

How to read this map: the top row builds the virtual world; the box at the bottom is the gate. The weak point is the cyan box: the vision model gets measurements wrong, so the test varies sizes and positions to avoid approving a plan that only works with perfect measurements.

Step by step with robotica-real2sim

Goal: run the project's ready-made example (with the Franka Panda arm from the MuJoCo collection) without depending on any external service.

git clone https://github.com/inematds/robotica-real2sim
cd robotica-real2sim
./instalar.sh                     # venv + MuJoCo + arm model + tests
# without a monitor (server, WSL): export MUJOCO_GL=egl  (without a GPU: osmesa)
./r2s.py filmar exemplos/mesa-teste/cena.json ep.mp4 --cameras geral,cima,pulso
./r2s.py episodios exemplos/mesa-teste/cena.json saida/ -n 50
./r2s.py provar exemplos/mesa-foto/cena.json resultados/mesa-foto/plano.json --sem-vagas
./r2s.py provar exemplos/mesa-foto/cena.json resultados/mesa-foto/plano.json

Why learn this

The test result is the lesson: a plan that seemed obvious failed 16 times out of 20, and only the simulation revealed that, without knocking anything over. Building the scene from your own table is what turns simulation from a polished demo into a real tool.

Key concepts

Every simulation is an approximation. The difference between what happens in the digital twin and the real arm has a name: sim2real gap (in English, reality gap). It comes from friction, servo backlash and delay, the actual mass of the parts, and the light and texture seen by the camera. You can't eliminate this gap; you measure it and train the policy to handle it.

How to read this map: The green curve (twin) rises smoothly to the target. The dashed red curve (real) starts late, overshoots, and stops with a small error, typical of a cheap servo with backlash. The blue band shows the set of sampled twins: if it covers the real curve, the policy trained on it has already “seen” something similar to the real world.

Goal: Run the same command on 20 versions of the SO-101, with sampled friction, servo force, and gripper mass, and measure how much the gripper position varies.

# Domain randomization: the same command, 20 "slightly different" arms.
# If the results vary a lot, your control depends on details that the real world doesn't guarantee.
import sys
import numpy as np
import mujoco

CENA = sys.argv[1] if len(sys.argv) > 1 else "mujoco_menagerie/robotstudio_so101/scene.xml"
base = mujoco.MjModel.from_xml_path(CENA)
rng = np.random.default_rng(42)
garra = base.body("gripper").id
finais = []

for k in range(20):
    model = mujoco.MjModel.from_xml_path(CENA)
    model.dof_damping[:] = base.dof_damping * rng.uniform(0.7, 1.3)          # internal friction ±30%
    model.actuator_gainprm[:, 0] *= rng.uniform(0.8, 1.2)                    # servo force ±20%
    model.actuator_biasprm[:, 1] = -model.actuator_gainprm[:, 0]            # keeps the position servo consistent
    model.body_mass[garra] = base.body_mass[garra] * rng.uniform(0.8, 1.5)  # heavier gripper
    data = mujoco.MjData(model)
    data.ctrl[model.actuator("shoulder_lift").id] = -0.4
    data.ctrl[model.actuator("elbow_flex").id] = 0.6
    for _ in range(int(0.3 / model.opt.timestep)):   # just 0.3 s: the arm is still moving into position
        mujoco.mj_step(model, data)
    finais.append(data.xpos[garra].copy())

finais = np.array(finais)
print("average gripper position (m):", np.round(finais.mean(axis=0), 3))
print("spread (standard deviation, mm):", np.round(finais.std(axis=0) * 1000, 1))

Why learn this

If you ignore the sim2real gap, you may think the simulation “lied” when the real arm fails. If you measure it, you know in advance where it will fail and have already trained for that.

Key concepts

Recording 50 demonstrations by hand is tiring; generating 1,000 in simulation takes little computer time. This synthetic data provides variety (positions, sizes, masses), and a few real episodes correct the physics the simulation got wrong. The most common approach today is a mix: lots of synthetic data + some real data.

How to read this map: The green grid on the left (synthetic) is much larger than the blue line (real), but both go into the same training run. On the right are the three policies you compare on the real arm. If “real + simulation” beats “real only” with the same real episodes, the twin is helping.

Goal: Generate episodes with cameras in robotica-real2sim, export them in LeRobot format, and train an ACT directly from the local folder.

# inside the robotica-real2sim folder
./instalar.sh --lerobot           # creates .venv-lerobot just for the exporter
./r2s.py episodios exemplos/mesa-foto/cena.json sint/episodios -n 50 --imagens
./r2s.py exportar sint/episodios sint/lerobot

# train using the local folder (check the flags with: lerobot-train --help)
.venv-lerobot/bin/lerobot-train \
  --dataset.repo_id=local/real2sim --dataset.root=sint/lerobot \
  --policy.type=act --policy.device=cuda \
  --output_dir=outputs/train/act-sint --policy.push_to_hub=false

Why learn this

Data is the bottleneck in AI robotics. Knowing how to generate data in simulation and prove that it helps by comparing three versions is the skill that separates a science fair project from one that improves with every round.

Key concepts