🧠 Modern AI and robotic arm
Language models, voice, and vision on the robot, with safety built in: the model chooses, the code checks, the microcontroller executes. Then, an SO-101 arm that learns tasks by watching you demonstrate them, also trained in simulation. Finally, an assistant robot that listens to a request, plans, and picks up an object.
How to read this map: The first three steps (teal) build the pieces: a safe brain with a local model, offline voice, and a working SO-101 arm. The next two (cyan) teach the arm: through demonstrations and in simulation. The final step, with a glow, brings everything together in the assistant. If you already know a step, skip to the next one.
🦾 What you’ll build in this track
An assistant that listens to “pick up the red cube,” understands the request with a local language model, checks the action against a fixed list, and uses an SO-101 arm you trained to pick up the object. Along the way, you’ll build a brain with Ollama that accepts only permitted actions, a Portuguese voice pipeline that works offline, a calibrated leader/follower pair, and a policy trained on your demonstrations.
- ✓Local models: no cost per request
- ✓Layered safeguards and a physical stop
- ✓LeRobot with checked commands
- ✓Simulation before the real arm
💡 Don’t have the arm yet?
Modules 5.1 and 5.2 run on a regular PC, with the ESP32 from Track 1 or in simulation mode. Module 5.5 trains the SO-101 digital twin in simulation. Buy the arm when you reach 5.3.
Track map
🧠 Language model as the brain
The model chooses, the robot executes
🎙️ Local voice: listen and speak
Whisper and Piper, offline
🦾 SO-101 arm and LeRobot
Calibrate and teleoperate the pair
🎓 Learning by demonstration
You show it, the arm repeats
🧪 Simulate before going real
MuJoCo, real2sim, and sim2real
🦾 Project: assistant robot
Voice, plan, vision, and arm working together
Detailed content
🧠 Language model as the brain
LLM, VLM, and VLA on the robot; the architecture where the model only chooses actions from a fixed list; Ollama running locally; and safeguards that protect the motor.
Text to text, image with text to description, and image with instruction to movement: three model types, each in a different place in the robot stack.
Avoid asking a text model to control a motor in real time.
Parameters · inference · token · VLA.
The model stays at the high level; a code validator checks the action before it becomes a command over serial.
Models can make mistakes confidently; keeping these steps separate prevents mistakes from reaching the motor.
High level · validator · hallucination · final safeguard.
A menu of actions with parameters and limits, provided to the model as JSON Schema.
Turn an open-ended response into a closed, testable problem.
JSON Schema · enum · limits · tool use.
Ollama on localhost and a complete Python script that asks, validates, and sends commands to the ESP32 (with simulation mode).
Works without internet and has no per-request cost; cloud models also work with the same validator.
/api/chat · format · temperature 0 · pyserial.
Take a picture and ask the local vision model what’s in the scene.
Gives context to the planner without handing it control of the motor.
images in base64 · context · detector vs. VLM.
Four layers: validator, limits repeated on the ESP32, command watchdog, and physical stop.
Any model you plug in later operates within a box you control.
Layers · watchdog · confirmation · power switch.
🎙️ Local voice: listen and speak
Offline voice pipeline in Portuguese: Whisper listens, the 5.1 brain decides, and Piper responds, with a wake trigger, spoken confirmation, and measured latency.
Four separate stages: microphone, speech recognition, brain, and speech synthesis.
Testing each stage on its own shows where errors and delays occur.
STT · TTS · VAD · 16 kHz mono.
Record from the microphone and transcribe in Portuguese with faster-whisper, on CPU or GPU.
If recognition gets it wrong, everything else goes wrong too.
Model size · int8 · language="pt" · VAD.
Press to talk or say “hey, robot,” and where to place the microphone to avoid motor noise.
A robot that obeys any phrase in the room is unpredictable.
Trigger · openWakeWord · state machine · noise.
The complete Python assistant: listen, choose, validate, confirm by voice, execute, and respond.
Voice is just another input; the validator stays the same.
Import modules · spoken confirmation · fail-safe.
Lightweight speech synthesis with pt_BR voices that runs on Pi 5 without a GPU.
A robot that tells you what it’s going to do gives you time to stop it if that’s not what you wanted.
.onnx voice · phoneme · piper-tts · audio output.
Measure the time for each stage and decide what can leave your home.
You optimize the real bottleneck and avoid issues with students’ voices in the classroom.
Bottleneck · offline · voice data · consent.
🦾 SO-101 arm and LeRobot
The leader/follower pair with Feetech STS3215 servos: IDs, calibration, LeRobot installation, first teleoperation, cameras, and the safety routine.
An open-source, 3D-printed arm pair and the Hugging Face library that covers everything from motors to policies.
It’s the shortest, cheapest path to an arm that can learn a task.
Degree of freedom · leader/follower · STS3215 · policy.
Find the ports, write each servo’s ID, assemble the arm, and measure the limits of each joint.
Wrong IDs or poor calibration cause most cases of an arm shaking or moving in the wrong direction.
Bus · find-port · setup-motors · calibrate.
Virtual environment, feetech extra, USB port permissions, and a recorded version.
AI robotics libraries change quickly; isolating the environment and pinning the version keeps the project running.
venv · extras · dialout · --help.
You move the leader and the follower copies it, in a loop running dozens of cycles per second.
The 5.4 policy imitates exactly how you teleoperate.
Control loop · target position · initial pose.
Two fixed cameras, steady lighting, a simple background, and marked positions on the table.
The camera is the policy’s main sensor; changes in the scene confuse the model.
find-cameras · index_or_path · range · display_data.
Clear area, power switch within reach, fixed base, and a routine before each session.
A trained policy may make a movement you didn’t expect.
Physical stop · torque · pinch point.
🎓 Learning by demonstration
Record episodes with the SO-101 pair, publish the dataset, train an ACT policy, evaluate it on the real arm, and explore policies that understand language.
Instead of programming the movement, you demonstrate the task several times and a model learns to imitate you.
Tasks like picking up varied objects are hard to program by hand and easy to demonstrate.
Imitation · episode · observation · action.
Teleoperate while LeRobot saves camera images and joint positions in episodes.
The quality and variety of the demonstrations determine the training result.
Episode · reset · variation · task description.
Store, view, and share recorded episodes in the LeRobot format.
Reviewing the data before training avoids wasting GPU time on poor demonstrations.
Dataset · Hub · visualization · data version.
Train the small, fast ACT policy on an NVIDIA GPU or in Colab.
It’s the first training run that fits a maker’s budget and works with just a few dozen episodes.
Policy · training steps · checkpoint · loss.
Run the trained policy on the follower arm with lerobot-rollout, safely, and count how many times it succeeds.
Measured success rate is the only honest way to say the arm has learned.
Success rate · attempts · common failures.
VLA models that receive a text instruction along with camera feeds: the small SmolVLA and the larger pi0 family.
The same policy can perform different tasks depending on the request.
VLA · fine-tuning · base model · GPU cost.
🧪 Simulate before going real
Why train in simulation, the SO-101 twin in MuJoCo, when Isaac is worth it, turning a photo of your table into a scene, and the gap between simulation and reality.
Test and train in a virtual world, where collisions break nothing and time can run faster.
It saves parts, human demonstrations, and scares at the workbench.
Simulator · physics · scene · cost of errors.
The lightweight, open simulator, with a model of the SO-101 that moves like the real arm.
It runs on a regular laptop and lets you test before you have the arm.
MuJoCo · digital twin · MJCF model · joints.
NVIDIA’s simulation platform, with realistic graphics and many robots running in parallel, requires a powerful NVIDIA GPU.
Knowing when the extra horsepower is worth it helps you avoid building a setup bigger than the problem requires.
Isaac Sim · Isaac Lab · parallelism · GPU requirements.
Build the simulated scene from photos of your workbench, with objects in their real positions.
The more closely the simulation matches your table, the more useful the training will be on the real arm.
Real2sim · reconstruction · scene · object.
What changes when the policy moves from simulation to the arm: friction, play, lighting, camera.
Understanding the gap helps you reduce it instead of blaming the model.
Sim2real · domain randomization · real-world fine-tuning.
Automatically create episodes in simulation and combine them with real demonstrations.
This reduces the number of human demonstrations you need to record.
Synthetic data · real + simulated mix · comparison.
🦾 Final project: assistant robot
From defining the scope and rules to turning voice into a plan, using vision and an arm to pick up objects, integrating a mobile base, measuring success, and presenting the project.
Define what the assistant does, what it refuses, and the safety rules before writing code.
A small, well-defined scope is what makes a final project work on presentation day.
Scope · requirements · action list · completion criteria.
The spoken request becomes a short sequence of actions from a fixed list, checked step by step.
Real requests have multiple steps; the entire plan needs validation and must run with pauses.
Plan · sequence · validation · confirmation.
The camera finds the requested object, and the policy trained in 5.4 makes the arm pick it up.
This is where perception and action need to agree in the real world.
Detection · policy · pick and place · verification.
Combine the mobile robot from earlier learning paths with the arm, accounting for power, weight, and communication.
Integrating systems that work on their own is where real problems show up.
Integration · center of mass · separate power · states.
A repeated test plan, success rate for each step, and a record of failures.
The numbers show what to improve and keep you from relying on a video that only worked once.
Success rate · time per task · failure log.
Publish the README, a real-time video, the dataset, and the code so others can reproduce the project.
A documented project becomes a portfolio piece and helps the next person get started.
README · honest demonstration · license · community.
🧰 Trail kit
What the learning path uses beyond the robot from earlier paths. Approximate range of R$ 1.800 a R$ 4.500 not including the computer (Oct/2026; check before buying: the arm is usually imported, and the price varies widely with assembly, shipping, and exchange rates).
SO-101 leader-follower pair: 12 Feetech STS3215 servos, two serial servo adapter boards, power supplies, and 3D-printed parts (print them yourself or buy the kit).
One or two standard USB cameras (720p or 1080p), with a fixed mount: one at the front and another at the side or on the wrist.
A basic USB or lapel microphone and a USB or 3.5 mm speaker.
A PC with an NVIDIA GPU, or Google Colab, to train policies in 5.4. Modules 5.1 and 5.2 run on a regular PC.
Raspberry Pi 5 with 8 GB; with AI HAT+ 2 (about US$ 130) for small LLMs and VLMs; or Jetson Orin Nano Super (US$ 399 as of Jul/2026), which is more powerful and more expensive.
An on/off switch in series with the servo power supply, a clamp to secure the arm’s base to the table, and safety glasses.
✓ Worthwhile optional item
Access to a 3D printer (for replacement parts and camera mounts), a second camera, and a fixed workbench light.
✗ Don’t buy these for this trail
Hobby servo arm with PWM signal (SG90, MG996R): LeRobot cannot read their position. And don’t pay for a GPU without CUDA expecting to train policies.