Simulation twin of the SO-101: picks up, carries and drops light objects in physics, generates training data and proves the plan before a single motor is powered on.
It is a virtual copy (a "twin") of the SO-101 robotic arm, the low-cost 5-joint kit used with LeRobot, running in the MuJoCo physics simulator. It is for anyone who is going to build the arm and wants to test first: the twin picks up a cube, a tennis ball and a small pot and puts them in a tray, records the movements as training data and shows whether a plan works. All you need is a computer with Linux and Python; the real arm comes in the later phases.
It continues robotica-real2sim, which did the same with a Franka Panda industrial arm. The SO-101 reaches half as far and carries 15 times less, so almost everything changed: kinematics, gripper, table and objects.
Official SO-101 model (MuJoCo Menagerie): 5 joints in the arm, a gripper with one fixed finger and one that rotates, servos with a maximum torque of 2.94 NĀ·m.
A 3 cm cube, a tennis ball and a 4 cm pot go into the tray in physics: 50 out of 50 episodes for each one.
Every plan runs dozens of times with variations in position and size. Nothing goes to the real arm without passing the worst-case test.
The table becomes a cena.json file (written by hand or extracted from a photo by a vision model). From there on, everything runs locally, with no API.
The tip position rules; the gripper points straight down when possible and tilts on its own when the arm cannot reach. The wrist rotation chooses which side the gripper closes on.
The object is grasped against the fixed finger; the ball is grasped slightly below its center, otherwise the moving finger pushes it down and it slips.
The carried object passes only 1.5 cm above the tray rim and goes around whatever is on the table: going over a tennis ball is already out of reach.
Tested on Linux with Python 3.12. No dedicated graphics card needed: images are rendered through EGL.
The installer creates the virtual environment and downloads the SO-101 model.
git clone https://github.com/inematds/robotica-so101 cd robotica-so101 && ./instalar.sh
Only for the filmar command.
sudo apt install ffmpegOnly to export the episodes as a LeRobot dataset.
./instalar.sh --lerobotAll commands below run in the project folder. Always use MUJOCO_GL=egl if there is no display (r2s.py already does this).
The tests check the scene, the gripper reach, the plans and the episodes.
./instalar.sh python3 testes/rodar.py # 14 cases -> TUDO OK
The example scene has a cube, a tennis ball, a small pot and a tray, 15 to 30 cm from the base. Cameras: geral (overview), cima (top), frente (front) and pulso (wrist).
./r2s.py ver exemplos/mesa-so101/cena.json mesa.png --camera geral
A video of the scene's task (pick up the cube and put it in the tray), with three cameras side by side.
./r2s.py filmar exemplos/mesa-so101/cena.json so101.mp4 --cameras geral,cima,pulso
The plan is a list of pick ā destination steps. The test runs N times, each time with a slightly different table, and reports which step fails and why. Slots in the tray are computed from the size of each object.
./r2s.py provar exemplos/mesa-so101/cena.json resultados/mesa-so101/plano.json -n 20 ./r2s.py provar exemplos/mesa-so101/cena.json resultados/mesa-so101/plano.json -n 20 --pior-caso
Each episode randomizes the position, rotation and mass of the objects, records joints and commands at 50 Hz and a 4-level annotation (goal, subtask, primitive, events). With --imagens, it also records the cima and pulso cameras.
./r2s.py episodios exemplos/mesa-so101/cena.json /tmp/ep -n 50 # 50/50 in ~9 s ./r2s.py exportar /tmp/ep /tmp/ds # LeRobot dataset (requires --lerobot)
A vision model via subscription (Codex or Claude) reads the photo of the table and returns the objects in centimeters; the script brings everything within the SO-101's reach and warns about anything that exceeds the gripper (9 cm) or the payload (200 g).
./r2s.py foto foto.jpg cena.json --motor codex
Numbers measured on the exemplos/mesa-so101 scene, seed 0. The video has subtitles in Portuguese.
| Test | Result |
|---|---|
| 3 cm cube to the tray | 50/50 episodes |
| Tennis ball (6.7 cm, 58 g) | 50/50 episodes |
| 4 cm pot | 50/50 episodes |
| All three in the same tray, correct measurements | 18/20 |
| All three, size off by Ā±15% | 18/20 |
| All three, worst case (tray 15% smaller) | 6/20: the three barely fit |
| Cube + pot without slots Ć with slots | 0/6 Ć 6/6 |
So as not to promise what the project does not do.
The real arm has not been powered on yet. The driver for the Hiwonder HX servos in LeRobot has not been confirmed yet.
Laying a tall can down in a shallow tray exists in the Panda project, but has not been ported to the SO-101.
In the simulator the generator knows where each object is. In the real world, that position will come from the camera (Phase 4).
Each phase has a "command ā expected output" criterion in docs/PLANO.md (in Portuguese).
