🎯 Project: a robot that follows an object

The robot wants two things: to keep the target in the center of the image and at a fixed distance. The camera doesn't measure distance, but the object's apparent size is useful: nearby, it takes up many pixels; far away, it takes up few. So each frame becomes two error values, and each error controls a movement: the horizontal error tells it to turn, the size error tells it to move forward or backward.

How to read this map: the four purple boxes run on the Pi, once per frame; the cyan box is the ESP32. The error is highlighted because that's where the image becomes a number. If something goes wrong, find which box: detection doesn't find the target, the error has the wrong sign, or the motor doesn't respond.

Why learn this

Following with a camera uses the same sense-think-act loop as module 1.1, except the sensor is now an image and “think” includes computer vision. This setup (detector → error → controller → actuator) also works for a tracking security camera, a drone that follows something and an arm that aims.

Key concepts

Start with the Track 1 rover (chassis, motors, TB6612FNG and ESP32). Add: the Pi 5 with active cooler, the Camera Module 3 at the front and a separate power supply for the Pi. The Pi 5 needs 5 V at up to 5 A (the official power supply is 27 W); on the robot, use a 5 V, 5 A buck converter connected to a 2S pack (two 18650 cells) with a BMS. The motors use their own battery, and all GNDs are connected.

How to read this map: the camera goes in the center at the front, because the code assumes the center of the image is the front of the robot. Purple powers the Pi; amber powers only the H-bridge. Keeping the two power supplies separate prevents a motor current spike from restarting the Pi. The dashed line is the UART cable from module 3.5.

Step by step

Why learn this

Half of all Pi robot “bugs” are power issues: random restarts, corrupted cards, cameras that disappear. Using separate power supplies, a common GND, and measuring before powering on gets rid of these ghosts before you even open the code.

Key concepts

Start with the lightest version: a brightly colored ball (blue, for example), detected using the HSV mask and largest contour, just as in module 3.2. At 320×240, the Pi 5 processes dozens of frames per second, making control smooth. Here, the control is just proportional (speed = gain × error); in topic 5, it becomes PID.

How to read this map: The dashed line marks the center of the image, which aligns with the front of the robot. The purple arrow shows the error: it points in the direction the robot needs to turn. With a positive error, the left wheel spins faster than the right, and the robot turns right, bringing the ball back to the center.

Goal: The robot turns until the ball is centered, then moves toward it until it reaches the desired size. Save this in the same folder as link_esp32.py and run it in the course venv (created with --system-site-packages so it can find picamera2). Adjust HSV_MIN/HSV_MAX for your color.

#!/usr/bin/env python3
"""Robot that follows a colored object (module 3.6). Requires link_esp32.py (module 3.5) in the same folder."""
import time
import cv2
import numpy as np
from picamera2 import Picamera2
from link_esp32 import Link

LARGURA, ALTURA = 320, 240
HSV_MIN = np.array([100, 120, 70])    # blue range; calibrate with module 3.2
HSV_MAX = np.array([130, 255, 255])
AREA_MIN = 300                        # smaller than this is noise
AREA_ALVO = 6000                      # area (pixels) at the desired distance: measure yours
KP_GIRO, KP_DIST, VEL_MAX = 180, 160, 180


def acha_objeto(quadro):
    hsv = cv2.cvtColor(quadro, cv2.COLOR_BGR2HSV)
    mascara = cv2.inRange(hsv, HSV_MIN, HSV_MAX)
    mascara = cv2.morphologyEx(mascara, cv2.MORPH_OPEN, np.ones((5, 5), np.uint8))
    contornos, _ = cv2.findContours(mascara, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    if not contornos:
        return None
    maior = max(contornos, key=cv2.contourArea)
    area = cv2.contourArea(maior)
    if area < AREA_MIN:
        return None
    m = cv2.moments(maior)
    return m["m10"] / m["m00"], area      # center x and area


def comando(cx, area):
    erro_x = (cx - LARGURA / 2) / (LARGURA / 2)                    # -1 left .. +1 right
    erro_d = max(-1.0, min(1.0, (AREA_ALVO - area) / AREA_ALVO))   # >0 far, <0 close
    giro, avanco = KP_GIRO * erro_x, KP_DIST * erro_d
    esq = max(-VEL_MAX, min(VEL_MAX, avanco + giro))
    drt = max(-VEL_MAX, min(VEL_MAX, avanco - giro))
    return esq, drt


def main():
    cam = Picamera2()
    cam.configure(cam.create_video_configuration(main={"size": (LARGURA, ALTURA), "format": "RGB888"}))
    cam.start()
    link = Link()
    t0, quadros, esq, drt = time.monotonic(), 0, 0, 0
    try:
        while True:
            quadro = cam.capture_array()     # "RGB888" from picamera2 is already in OpenCV's BGR order
            achado = acha_objeto(quadro)
            esq, drt = comando(*achado) if achado else (0, 0)   # target lost: stop
            link.motores(esq, drt)           # send a command every frame: keeps the ESP32 alive
            quadros += 1
            if time.monotonic() - t0 >= 1.0:
                print(f"{quadros} FPS | target={'yes' if achado else 'no'} | left={esq:.0f} right={drt:.0f}")
                t0, quadros = time.monotonic(), 0
    except KeyboardInterrupt:
        pass
    finally:
        link.fechar()                        # send "S"
        cam.stop()


if __name__ == "__main__":
    main()

Why learn this

The color-based version is fast, inexpensive, and easy to debug, so it's the best way to get the mechanics, signals, and gains right. Once it works, switching the detector to a neural network means changing one function.

Key concepts

To follow a person, color alone isn't enough. Replace the detector with YOLO26n exported to NCNN (module 3.3). Instead of area, use the box height as a measure of distance: it changes less when the person spreads their arms. The detector is slower than the color mask (a few frames per second on the CPU), so the gains here are lower. With the AI HAT+, the FPS increases considerably.

Goal: follow the largest person in the image. First, export the model at the size used here: yolo export model=yolo26n.pt format=ncnn imgsz=320 (creates the folder yolo26n_ncnn_model).

#!/usr/bin/env python3
"""Follows a person with YOLO26 (NCNN) on the Pi 5 (module 3.6). Requires link_esp32.py and the exported model."""
import time
from picamera2 import Picamera2
from ultralytics import YOLO
from link_esp32 import Link

LARGURA, ALTURA = 640, 480
CLASSE = 0              # 0 = "person" in COCO; 32 = "sports ball"
CONF_MIN = 0.5
ALTURA_ALVO = 0.6       # box height ÷ image height at the desired distance
KP_GIRO, KP_DIST, VEL_MAX = 140, 150, 150
modelo = YOLO("yolo26n_ncnn_model")


def maior_caixa(resultado):
    melhor = None
    for caixa in resultado.boxes:
        if int(caixa.cls) != CLASSE or float(caixa.conf) < CONF_MIN:
            continue
        x1, y1, x2, y2 = caixa.xyxy[0].tolist()
        if melhor is None or (y2 - y1) > (melhor[3] - melhor[1]):
            melhor = (x1, y1, x2, y2)
    return melhor


def main():
    cam = Picamera2()
    cam.configure(cam.create_video_configuration(main={"size": (LARGURA, ALTURA), "format": "RGB888"}))
    cam.start()
    link = Link()
    try:
        while True:
            t = time.monotonic()
            quadro = cam.capture_array()
            caixa = maior_caixa(modelo.predict(quadro, imgsz=320, verbose=False)[0])
            esq = drt = 0
            if caixa:
                x1, y1, x2, y2 = caixa
                erro_x = ((x1 + x2) / 2 - LARGURA / 2) / (LARGURA / 2)
                erro_d = max(-1.0, min(1.0, (ALTURA_ALVO - (y2 - y1) / ALTURA) / ALTURA_ALVO))
                giro, avanco = KP_GIRO * erro_x, KP_DIST * erro_d
                esq = max(-VEL_MAX, min(VEL_MAX, avanco + giro))
                drt = max(-VEL_MAX, min(VEL_MAX, avanco - giro))
            link.motores(esq, drt)
            print(f"{(time.monotonic() - t) * 1000:.0f} ms | person={'yes' if caixa else 'no'} | left={esq:.0f} right={drt:.0f}")
    except KeyboardInterrupt:
        pass
    finally:
        link.fechar()
        cam.stop()


if __name__ == "__main__":
    main()

Why learn this

Changing the color detector to a neural network shows the value of a layered architecture: the link, the ESP32, and the wheel logic stayed the same. After training YOLO26 with your own images (module 3.3), the same program can follow your object, the school mascot, or another robot.

Key concepts

With only the P term, the robot tends to overshoot and wobble from side to side (camera delay makes this worse). The D slows the turn as the error decreases; the I corrects for the small remaining distance caused by friction. These are two independent PIDs, as in module 2.3: one for turning and one for distance, combined when sending commands to the wheels.

How to read this map: each error has its own controller, and the crossing arrows show that both wheels receive both components. Turning has the opposite sign for each wheel; forward motion has the same sign. Tune one PID at a time: first turning with forward motion set to zero, then distance.

Goal: a reusable PID with an output limit and windup protection (the integral stops growing when the output saturates).

"""Simple PID with limit and anti-windup (module 3.6)."""
import time


class PID:
    def __init__(self, kp, ki=0.0, kd=0.0, limite=255.0):
        self.kp, self.ki, self.kd, self.limite = kp, ki, kd, limite
        self.reset()

    def reset(self):
        self.integral, self.erro_ant, self.t_ant = 0.0, 0.0, None

    def __call__(self, erro):
        agora = time.monotonic()
        dt = 0.0 if self.t_ant is None else agora - self.t_ant
        deriv = (erro - self.erro_ant) / dt if dt > 0 else 0.0
        self.integral += erro * dt
        saida = self.kp * erro + self.ki * self.integral + self.kd * deriv
        if abs(saida) > self.limite:          # saturated: undo the sum (anti-windup)
            self.integral -= erro * dt
            saida = max(-self.limite, min(self.limite, saida))
        self.erro_ant, self.t_ant = erro, agora
        return saida

In seguidor_cor.py, add the import and the two objects after the constants, then replace the entire command function with this one. In main(), when the target disappears, call pid_giro.reset() e pid_dist.reset().

from pid import PID

pid_giro = PID(kp=180, ki=0.0, kd=20, limite=150)   # start with kd=0 and increase it gradually
pid_dist = PID(kp=160, ki=30, kd=0, limite=150)


def comando(cx, area):
    erro_x = (cx - LARGURA / 2) / (LARGURA / 2)
    erro_d = max(-1.0, min(1.0, (AREA_ALVO - area) / AREA_ALVO))
    if abs(erro_x) < 0.05:                    # dead zone: don't jitter when the target is centered
        erro_x = 0.0
    giro, avanco = pid_giro(erro_x), pid_dist(erro_d)
    esq = max(-VEL_MAX, min(VEL_MAX, avanco + giro))
    drt = max(-VEL_MAX, min(VEL_MAX, avanco - giro))
    return esq, drt

How to check: with the robot on the floor, push the ball to one side. With only P, it overshoots and turns back; with D tuned, it reaches the center with at most a small overshoot. If it starts to jitter, kd is too high.

Why learn this

The same PID as the line follower in Trail 2, now with a slow, noisy sensor (the camera). Learning to tune with delay is what separates a robot that follows smoothly from one that dances around the target.

Key concepts

Test in stages, from safest to most realistic: wheels off the ground, clear floor with a low maximum speed (VEL_MAX = 100), and only then the final speed. For each run, note what changed and the result, as in the Trail 2 logbook. The most common problems have known causes:

Improvements (challenges)

Why learn this

A project is finished when it works repeatedly and safely, not the first time it succeeds. Documenting symptoms, causes, and adjustments makes it possible to present the robot, teach with it, and keep improving without starting over.

Key concepts