👁️ Embedded vision and AI
The robot starts to see. You add a Raspberry Pi 5 with a camera to the chassis, detect objects by color with OpenCV and by shape with YOLO26, run small AI models on the ESP32-S3 itself, and have the Pi communicate with the ESP32. By the end, you’ll build a robot that follows an object or a person.
How to read this map: The first three steps (purple) build vision on the Raspberry Pi 5: the operating system, camera, and detector. The next two (cyan) bring AI onto the ESP32 and connect the two boards. The final step, with a glow, brings everything together in the robot that follows. If you already use Linux, start at 3.2.
🎯 What you’ll build in this track
The Track 1 rover gets a Raspberry Pi 5 with a camera. The Pi finds the target, works out which way it is and how far away it is, then sends short commands to the ESP32, which turns the wheels. First, the robot follows a colored ball; then a person or an object you taught it to recognize. Along the way, you’ll build a live video feed of the robot in your browser, a detector running in NCNN, and an ESP32-S3 that recognizes a gesture on its own.
- ✓Raspberry Pi OS 64-bit (Trixie) and picamera2
- ✓Ultralytics YOLO26, the current detector
- ✓Pi thinks, ESP32 acts, with a safe stop
- ✓A direct foundation for the ROS 2 track in Track 4
💡 Don’t have a Pi 5 yet?
Section 3.4 runs with an ESP32-S3 alone. OpenCV and YOLO26 also work on your PC with a webcam, so you can learn about masks and detection before buying the board.
Track map
🍓 Raspberry Pi 5 and Linux
A whole computer inside the robot
📷 Camera and OpenCV
Find the object by its color
🔍 Detect objects with YOLO26
A person, bottle, or your own part
🧬 AI inside the microcontroller
A small model, no computer needed
🔗 Raspberry Pi communicates with the ESP32
One thinks, the other acts
🎯 Project: a robot that follows an object
Vision closes the loop
Detailed content
🍓 Raspberry Pi 5 and Linux
Why the robot needs a computer, setting up the Pi 5 without a monitor, the terminal, Python with venv, 3.3 V GPIO, and power and heat considerations.
How the work is split: the Linux-based Pi 5 processes the camera feed and makes decisions; the ESP32 controls the motors in real time.
Avoids trying to run vision on the microcontroller or connecting a motor directly to the Pi.
Single-board computer · Linux · real time · Pi thinks, ESP32 acts.
Flash Raspberry Pi OS 64-bit with Wi-Fi and SSH already configured, then connect to the Pi over the network without a monitor.
There’s no screen on the robot; a properly set up Pi appears on the network on its own.
Headless · Pi Imager · SSH · robo.local · apt.
The fifteen commands that take care of almost everything: navigate, edit, copy, check disk space and memory, and shut down safely.
The camera, YOLO26, and ROS 2 all run in the terminal.
Home directory ~ · sudo · nano · Ctrl+C · shutdown.
Create a venv with access to system libraries (picamera2, gpiozero) and install packages in it with pip.
The system blocks global pip installs; the right venv avoids “the script can’t find the camera.”
venv · pip vs. apt · --system-site-packages · requirements.txt.
Status LED and stop button with gpiozero, using BCM numbering.
5 V on a GPIO pin will burn out the Pi; the physical stop button is covered in project 3.6.
BCM vs. physical · LED and Button · pinout · 3.3 V maximum.
27 W power supply at the workbench, buck converter in the robot, active cooler, and a temperature and undervoltage monitor.
“Mysterious” restarts and slowdowns are almost always caused by power or heat.
5 V buck converter · undervoltage · throttling · get_throttled.
📷 Camera and OpenCV
Pixels and channels, Camera Module 3 with picamera2, HSV color masks, object center, FPS, and live video in the browser.
The image as a table of numbers: height × width × 3 channels, in OpenCV’s BGR order.
Swapped colors and an inverted y-axis cause half of vision errors.
Pixel · channel · BGR · NumPy shape.
Connect the camera with the Pi 5 cable, test with rpicam-hello, and capture frames in Python.
It’s the entry point to everything that follows in the learning path.
rpicam-apps · picamera2 · capture_array · 22-to-15-pin cable.
Convert to HSV, choose a color range, and create a mask for the object.
This is the lightest form of vision: dozens of frames per second, with no AI.
Hue, saturation, brightness · inRange · morphology · red in two ranges.
Find the largest contour, calculate its center and area, and turn that into horizontal error.
Error and area are the two numbers the robot uses to turn and move closer.
findContours · moments · error = cx − midpoint · minimum area.
Measure how many times per second the entire loop runs, then choose the resolution based on the numbers.
Low FPS means delayed reactions; without measuring, optimization is guesswork.
Loop FPS · latency · half the resolution = ¼ of the work.
A pure Python MJPEG server that shows you what the robot sees, with overlays, on your phone.
Viewing the camera helps you tell a vision error from a control error in seconds.
MJPEG · port · thread · overlay · local network only.
🔍 Detect objects with YOLO26
What object detection is, YOLO26 on the Pi 5 with Ultralytics, exporting to NCNN, AI HAT+ (Hailo), training with your own images, and the real limitations.
For each object, a box, a class, and a confidence score; different from classification or segmentation.
Recognizes objects by shape, not color: it can follow a person wearing any clothes.
Box · class · confidence · inference · COCO.
Install Ultralytics in the venv and run yolo26n live with the Camera Module 3.
Twenty lines of code and the robot recognizes people and everyday objects.
yolo26n.pt · sizes n to x · conf · r.boxes.
Convert the model to NCNN, the fastest format on the Pi’s CPU, and compare the results with measurements.
More speed at no extra cost, and the habit of measuring before and after.
yolo export · NCNN · imgsz · warm-up.
AI HAT+ (Hailo-8/8L) and AI HAT+ 2 (Hailo-10H): NPUs on the Pi 5’s PCIe that take the neural network off the CPU.
Knowing when an accelerator is worth it helps you avoid buying one too soon or sticking with the CPU for too long.
NPU · PCIe · TOPS · compiled .hef model.
Take photos, label them in YOLO format, train from yolo26n on a PC or Colab, and bring best.pt to the Pi.
The robot can now recognize the object in your project, not just objects in COCO.
Fine-tuning · dataset · training and validation · data.yaml · best.pt.
Where detection fails and how to reach a stable decision: threshold, minimum size, and several consecutive frames.
A robot that follows people is only safe if you know where vision can fail.
False positive · false negative · threshold · persistence.
🧬 AI inside the microcontroller
TinyML: small, quantized models running on the ESP32-S3, with Edge Impulse or TensorFlow Lite Micro, for gestures, keywords, and people.
Run a model on the microcontroller itself, without a Pi or internet, using very little power.
Small, inexpensive robots can react to gestures and sounds without an onboard computer.
TinyML · local inference · low power consumption · privacy.
How a model is shrunk to fit in hundreds of kB: fewer layers and 8-bit integers.
Explains what does and doesn’t fit on an ESP32-S3.
Quantization · int8 · RAM and flash memory · accuracy vs. size.
Collect data, train in the browser, and download an Arduino library ready for the ESP32.
It’s the shortest path from scratch to your first running model, without having to program the neural network.
Data collection · impulse · training · export library.
Google’s inference engine for microcontrollers, on the ESP32 via esp-tflite-micro and ESP-NN optimizations.
Gives you full control over the model when a ready-made tool isn’t enough.
LiteRT · TensorFlow Lite Micro · esp-tflite-micro · ESP-NN.
Use the IMU from Track 2 to have the robot recognize movements like “shake” or “tilt.”
It’s the most affordable TinyML example: you already have the sensor.
Data window · gesture classes · confidence threshold.
Listen for a spoken command with a microphone or detect a person with an ESP32-S3 with a camera.
Shows the microcontroller’s limits and when to move to the Pi.
Keyword · person / no person · XIAO ESP32-S3 Sense.
🔗 Raspberry Pi communicates with the ESP32
UART between the boards, a simple message protocol with checks, both sides of the code, and what to do when the link drops.
Who decides what: the Pi sends short intentions; the ESP32 carries them out and enforces safety limits.
A clear boundary makes each board easy to test on its own.
High level vs. low level · command · limit on the microcontroller.
Connect TX to RX and RX to TX, with a shared GND. Power both boards at 3.3 V, or use the USB cable.
A swapped wire or missing GND is the number one cause of “nothing comes through.”
TX/RX crossed · baud rate · shared GND · serial port.
Use one message format per line, easy to read, with validation to discard anything received corrupted.
A partial message must never make a motor start spinning.
Text line · short JSON · checksum · acknowledgment response.
Read serial without blocking the loop, validate each command, and limit speeds on the ESP32 itself.
The microcontroller is the last line of defense before the motor.
Line buffer · validation · limits · millis().
Open the serial port in Python, send commands at a fixed rate, and read responses from the ESP32.
It bridges the detector in 3.3 and the wheels.
pyserial · timeout · send rate · read response.
What happens when the Pi freezes, the cable comes loose, or a board restarts, and how the robot stops on its own.
A safe robot stops when communication is lost.
Command watchdog · heartbeat · reconnection · safe state.
🎯 Project: a robot that follows an object
Camera, detection, error, and motors working together: first following a color, then a person or object with YOLO26, with PID and safety tests.
The diagram of the whole robot and what each module in the course provides.
With the plan on paper, every fault has a clear place to investigate.
Vision-based closed loop · stages · interfaces between boards.
Where to mount the camera, Pi, ESP32, buck converter, and battery, with short wiring and ventilation.
A crooked camera and poorly distributed power can ruin the best code.
Camera height · power branches · shared GND · mounting.
The robot’s first version: the HSV mask from 3.2 generates the error that turns the wheels.
It’s fast and lightweight, and validates the whole pipeline before you bring in AI.
Horizontal error · area as distance · lost target = stop.
Replace the mask with the detector from 3.3, using a stable decision across multiple frames.
See how the same controller can accept smarter vision without changing.
Bounding box → error · box size · NCNN · persistence.
Two controllers: one reduces horizontal error to zero (turning), and the other maintains distance (moving forward).
Without PID, the robot zigzags or runs into the target.
Trail 2 PID · dead zone · saturation · fine-tuning.
A test plan with the robot raised and on the floor, an emergency stop, and ideas for what to try next.
A robot that follows people needs to prove it stops when it should.
Test while raised · stop button · minimum distance · next level (ROS 2).
🧰 Trail kit
Beyond the Trail 1 rover. The basic setup (Pi 5, power supply, card, cooler, and camera) costs approximately R$ 1.100 a R$ 1.900 (Oct. 2026; check before buying: prices vary widely by store, shipping, exchange rate, and Pi memory). Optional items cost extra.
Raspberry Pi 5 with 4 GB (enough for this trail) or 8 GB (more headroom for YOLO26 and ROS 2). The Pi 4 works for the basics, but it doesn’t have the AI HAT+ PCIe interface.
Official 27 W power supply (USB-C) for your workbench, official active cooler, and, on the robot, a 5 V buck regulator rated for 5 A or more.
A microSD card with at least 32 GB from a reputable brand (class A1/A2). Ultralytics and the models take up several GB.
Camera Module 3 with the cable for Pi 5 (22 → 15 pins). The AI Camera (Sony IMX500) is an alternative with AI built into the camera.
AI HAT+ (Hailo-8L/8) for vision, or AI HAT+ 2 (Hailo-10H), around US$ 130 at US retailers in 2026, which also runs small language models.
ESP32-S3 with a camera and microphone, such as the XIAO ESP32-S3 Sense, for module 3.4. The IMU from Trail 2 also works.
✓ Worthwhile optional item
USB card reader, M2.5 spacers to mount the Pi on the chassis, and a brightly colored ball or sheet of poster board for color tests.
✗ Don’t buy these for this trail
Generic phone charger, power bank without 5 A, unbranded card, and the old “AI Kit” (discontinued; replaced by the AI HAT+).