PTENES
← Back to Track 2 MODULE 2.3

🌐 Simulators: MuJoCo, Isaac Sim, Habitat

Real data is expensive and dangerous; simulation is cheap, safe, and parallelizable across thousands of environments. Here you compare MuJoCo/MJX, Isaac Sim/Lab, and Habitat, understand the sim-real gap, and measure generalization in LIBERO and Meta-World.

6
Topics
45
Minutes
Inter.
Level
Practice
Type

Detailed content

N parallel envs on the GPU → one policy env 1 env 2 env 3 … env N Policy batched rollout Update grad / RL
1

🔹 Why simulate: cheap, safe, parallel, infinite resets

Every real demo requires a human, a robot, and clock time. In simulation, a cluster generates millions of trajectories overnight, with no mechanical wear or costly collisions. Crucial for RL, which needs orders of magnitude more interactions than imitation.

📊 The four advantages

  • Cost — no hardware wearing out, no human operator per trajectory.
  • Safety — bad policies break pixels, not actuators.
  • Parallelism — thousands of envs running simultaneously on the GPU.
  • Infinite reset — ground-truth state, programmatic randomization, replay.

⚡ Practical tip

Sim does not replace reality—it complements it. The winning recipe is yes for volume + RL e real for final fine-tuning with few demos. Trying to transfer 100% from sim without real data is usually frustrating.

Cost

Data is almost free.

Safety

Failure without physical damage.

Parallelism

Thousands of envs.

Ground truth

Exact state available.

2

🔹 MuJoCo / MJX: precise contact and differentiable physics

MuJoCo (Multi-Joint dynamics with Contact, now open-source from DeepMind) is a reference for contact and friction physics in manipulation. MJX reimplements the solver in JAX, running thousands of simulations on GPU/TPU with vmap and gradients via grad — enabling differentiable trajectory optimization.

pip install mujoco mujoco-mjx jax[cuda12]

import mujoco, mujoco.mjx as mjx, jax
model = mujoco.MjModel.from_xml_path("scene.xml")   # MJCF
mx = mjx.put_model(model)

@jax.vmap                       # 4096 mundos em paralelo
def step(state, ctrl):
    data = mjx.step(mx, state.replace(ctrl=ctrl))
    return data
# jax.grad(step) -> física diferenciável

⚡ Practical tip

Scenes in MuJoCo are MJCF (XML). Start with the models from mujoco_menagerie (Franka, ALOHA, hands) instead of modeling from scratch—incorrect collision geometry is the #1 cause of unstable physics.

MuJoCo

Precise contact.

MJX

JAX, vmap on GPU.

Differentiable

Dynamics gradient.

MJCF

XML scene.

3

🔹 NVIDIA Isaac Sim / Isaac Lab: GPU simulation with RTX

Isaac Sim (on Omniverse/PhysX 5) delivers physics and rendering RTX on the GPU; Isaac Lab is the RL/learning framework built on top of it, running thousands of parallel environments without leaving the GPU—observations, physics, and policy coexist in the same VRAM, eliminating the CPU↔GPU bottleneck.

📊 Why scale matters

  • PhysX 5 on GPU — thousands of envs, with no round trip to the CPU.
  • RTX — photorealistic rendering useful for visual sim-to-real.
  • USD/OpenUSD — an interoperable, modular scene format.
  • Sensors — ready-to-use synthetic cameras, LiDAR, and RGB-D.

✓ Do

  • ✓Use Isaac Lab for massive RL (locomotion, whole-body).
  • ✓Use RTX for visual domain randomization.
  • ✓Check NVIDIA GPU and driver requirements before getting started.

✗ Avoid

  • ✗Expecting Isaac Sim to run on non-NVIDIA hardware.
  • ✗Confusing photorealism with physical fidelity.
  • ✗Ignore setup costs; MuJoCo is lighter for prototyping.

Isaac Sim

Omniverse + PhysX 5.

Isaac Lab

RL framework.

RTX

Photorealistic render.

OpenUSD

Interoperable scene.

4

🔹 Habitat and related tools: navigation and embodied QA

Habitat (Meta) focuses on photorealistic 3D environments for navigation e embodied question answering — the agent moves through scanned scenes (HM3D, Matterport), answering "where is the key?". It’s another point on the spectrum: less contact manipulation, more spatial perception and exploration.

📊 Simulator spectrum

  • MuJoCo — contact/manipulation, fine-grained physics.
  • Isaac Sim — scale + RTX, locomotion, and manipulation.
  • Habitat — navigation, EQA, real scanned scenes.
  • Genesis / SAPIEN — new options 2024-26, speed and articulated robots.
import habitat_sim
cfg = habitat_sim.SimulatorConfiguration()
cfg.scene_id = "data/scene_datasets/hm3d/00800-TEEsavR23oF.glb"
sim = habitat_sim.Simulator(habitat_sim.Configuration(cfg, [agent_cfg]))
obs = sim.step("move_forward")   # navegação em cena escaneada real
obs["rgb"], obs["depth"]         # sensores sintéticos

Habitat

3D navigation.

EQA

Embedded questions.

HM3D

Scanned scenes.

SAPIEN

Articulated objects.

5

🔹 The physics and rendering gap (sim ≠ real)

No simulator is the real world. The reality gap appears with poorly modeled friction, deformable objects and liquids, sensor noise and latency, and visual domain differences (lighting, textures). A policy that gets 95% right in sim can plummet in the real world—a central theme of Track 3.

1

Physics

Friction, contact, deformables, and fluids are approximations; rigid objects are easy, cloth isn't.

2

Sensors

Real RGB-D has noise, holes, and latency that a perfect render ignores.

3

Appearance

Visual distribution differs; domain randomization mitigates this (Track 3).

⚡ Practical tip

Evaluate in the real world (or in a simulation different from the one used for training). Reporting only the success rate in the training simulation is the most common way to fool yourself about a policy.

Reality gap

Sim ≠ reality.

Deformables

Difficult fabrics/liquids.

Sensor noise

Imperfect RGB-D.

Honest evaluation

Measure real-world transfer.

✓ Do

  • ✓Randomize physics/visuals during training to close the gap.
  • ✓Set aside a sim/scenario for evaluation, separate from training.
  • ✓Validate on the real robot before declaring success.

✗ Avoid

  • ✗Report success rate only in the training sim.
  • ✗Model deformables/fluids as rigid and trust the results.
  • ✗Ignore real sensor noise/latency in the pipeline.
6

🔹 Benchmarks: LIBERO, Meta-World, RoboSuite

Benchmarks standardize comparisons. LIBERO measures continual learning and knowledge transfer in manipulation (LIBERO-Spatial, Object, Goal, Long); Meta-World offers 50 tasks for meta/multi-task RL; RoboSuite (MuJoCo) is a widely used modular manipulation suite in VLA papers.

⚡ Practical tip

Report the benchmark complete, not just the suite where your model shines. Cherry-picking a single LIBERO suite is robotics' "p-hacking"—reviewers and the community expect all four suites plus the average.

# avaliar OpenVLA fine-tunado em LIBERO (suites padrão)
python run_libero_eval.py \
  --model_path outputs/openvla-libero \
  --task_suite_name libero_spatial \
  --num_trials_per_task 50
# reporta success rate médio por suite -> número comparável entre papers

📊 What each one measures

  • LIBERO — lifelong learning, 4 transfer suites.
  • Meta-World — 50 tasks, ML1/ML10/ML45 for generalization.
  • RoboSuite — modular manipulation, the basis of many VLAs.

LIBERO

Lifelong / transfer.

Meta-World

50 multi-task tasks.

RoboSuite

Modular manipulation.

Success rate

Comparable metric.

✅ Module summary

✓
Simulate — inexpensive, safe, parallel, and infinitely resettable.
✓
MuJoCo/MJX — precise contact and differentiable physics in JAX.
✓
Isaac Sim/Lab — GPU and RTX scaling for massive RL.
✓
Reality gap — sim ≠ real; evaluate honestly.
✓
Benchmarks — LIBERO, Meta-World, RoboSuite compare policies.

Next module

2.4 — ROS2: the middleware that takes your policy from the simulator to the real arm, with tf2, MoveIt2, and real-time support.

← Back to the track Next module →