Detailed content
🖼️ How a computer sees an image
To a program, a photo isn’t a scene: it’s a table of numbers. Each little square, called a pixel, stores three values from 0 to 255, one for each primary color. A 640 × 480 image is a table with 480 rows, 640 columns and 3 layers. In Python, it becomes a NumPy array with the shape (480, 640, 3). Finding “the red ball” is really a matter of searching the table for a region where the numbers follow a certain pattern.
🌱 New here?
A channel is one of the image’s color layers. OpenCV stores colors in the order BGR (blue, green, red), unlike the RGB order you see in editors. NumPy is the Python library for tables of numbers; OpenCV is the computer vision library that works with those tables. Coordinate (0, 0) is the upper left corner: x increases to the right and y increases downward.
How to read this map: on the left, a section of an image enlarged until the little squares are visible. The pixel highlighted in purple is “red” because the R channel value is high and the others are low. Notice the B, G, R order: that’s what OpenCV uses. The cyan arrows show that y is counted from top to bottom.
Just one channel, from 0 (black) to 255 (white). Three times less data; great for edges and shapes.
Three channels, as delivered by the camera. Good for displaying and saving, poor for separating colors when the lighting changes.
Hue, saturation, and brightness. Separates “what color” from “how much light”: the right format for finding a colored object (topic 3).
Why learn this
When the mask misses the object or its center appears in the wrong place, the problem is almost always here: swapped colors (RGB treated as BGR), the y-axis inverted in the head, or an image that’s too large for the Pi to process in time.
Key concepts
📷 Camera Module 3 with picamera2
A Camera Module 3 connects to the Pi 5 camera connector with a flat cable (ribbon). In the terminal, the rpicam-apps (rpicam-hello, rpicam-still) test the camera in seconds. In your Python code, images are delivered by picamera2, which comes with Raspberry Pi OS. Older tutorials use command-line tools that are no longer available on the current system; if the command doesn’t start with rpicam-, be suspicious.
🌱 New here?
The Pi 5 connector is smaller (22 pins) than the camera connectors (15 pins): use the cable made for the Pi 5, which has one end of each size. Orient the metal contacts as indicated on the connector; open the latch with your fingernail, insert the cable straight, and close it. Do this with the Pi powered off.
How to read this map: from left to right, light turns into numbers. The ISP (image signal processor) corrects color and exposure before you receive the frame; that’s why the first image right after turning on the camera may come out dark, and the code waits a second before capturing.
Goal: confirm that the camera was detected, take a photo from the terminal, then capture a frame in Python. Install the system OpenCV once (the module 3.1 venv can see it).
rpicam-hello --list-cameras # should list the camera (imx708 = Camera Module 3)
rpicam-still -o ~/robo/teste.jpg # takes a photo from the terminal
sudo apt install -y python3-opencv # system OpenCV, visible in the venv
source ~/robo-venv/bin/activate# Captures a camera frame and saves it as JPEG
import time, os
import cv2
from picamera2 import Picamera2
picam2 = Picamera2()
config = picam2.create_preview_configuration(main={"size": (640, 480), "format": "RGB888"})
picam2.configure(config)
picam2.start()
time.sleep(1) # lets auto exposure adjust
quadro = picam2.capture_array() # NumPy array in BGR order
print("formato:", quadro.shape, quadro.dtype)
print("center pixel (B, G, R):", quadro[240, 320])
cv2.imwrite(os.path.expanduser("~/robo/quadro.jpg"), quadro)
picam2.stop()
print("saved to ~/robo/quadro.jpg")How to check
- ✓The terminal displays format: (480, 640, 3) uint8 and three numbers for the center pixel.
- ✓Copy the photo to your computer (scp <seu_usuario>@robo.local:~/robo/quadro.jpg .) and open it: the colors should look natural.
- ✗“no cameras available”? Turn it off, check the cable (orientation and latch) at both ends, and turn it back on.
Why learn this
This setup block (size, format, start, capture_array) is the entry point for everything that follows: color mask, YOLO26, and the course project. Testing first with the rpicam-hello separates hardware problems from code problems.
Key concepts
🎨 OpenCV: colors, HSV, and masks
A red ball in the shade and the same ball in the sun have very different BGR values. In the HSV color space, the difference is almost entirely in V (brightness), while H (hue, “what color”) stays similar. That’s why the standard approach is to convert the frame to HSV, choose an H, S, and V range, and create a mask, a black-and-white image in which only the pixels within the range appear white.
🌱 New here?
Hue (H) is the color on the color wheel; in OpenCV, it ranges from 0 to 179. Saturation (S) is how vivid the color is (0 = gray). Brightness (V) is how much light there is (0 = black). A mask is a filter: white where the condition is true, black everywhere else. Red lies at both ends of the wheel (near 0 and near 179), so it needs two ranges combined.
How to read this map: the range is the color wheel opened into a straight line, on OpenCV’s 0–179 scale. The purple frames show why red is tricky: it starts and ends the scale. The cyan frame shows a “well-behaved” color that fits in a single window.
Goal: generate a mask for a red object and save the frame and mask side by side, so you can adjust the range by looking at the result.
# HSV color mask (red = two ranges)
import time, os
import cv2, numpy as np
from picamera2 import Picamera2
picam2 = Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"size": (640, 480), "format": "RGB888"}))
picam2.start(); time.sleep(1)
quadro = picam2.capture_array()
picam2.stop()
hsv = cv2.cvtColor(quadro, cv2.COLOR_BGR2HSV)
baixo1, alto1 = np.array([0, 120, 70]), np.array([10, 255, 255]) # red, start of the scale
baixo2, alto2 = np.array([170, 120, 70]), np.array([179, 255, 255]) # red, end of the scale
mascara = cv2.inRange(hsv, baixo1, alto1) | cv2.inRange(hsv, baixo2, alto2)
# Cleanup: removes loose specks and fills holes
nucleo = np.ones((5, 5), np.uint8)
mascara = cv2.morphologyEx(mascara, cv2.MORPH_OPEN, nucleo)
mascara = cv2.morphologyEx(mascara, cv2.MORPH_CLOSE, nucleo)
brancos = cv2.countNonZero(mascara)
print(f"pixels in mask: {brancos} ({100 * brancos / mascara.size:.1f}% of image)")
lado_a_lado = np.hstack([quadro, cv2.cvtColor(mascara, cv2.COLOR_GRAY2BGR)])
cv2.imwrite(os.path.expanduser("~/robo/mascara.jpg"), lado_a_lado)
print("saved to ~/robo/mascara.jpg")How to check
- ✓With a red object in front of the camera, the right half of mascara.jpg shows a white patch in its shape.
- ✓Without the object, the percentage drops close to 0%.
- ✗Skin or wood showing up? Raise the minimum S (120 → 150). Object disappearing in the shade? Lower the minimum V.
✓ Helps color detection work
- ✓Bright, matte-colored object (ball, poster board, tape).
- ✓Background with a color very different from the object’s.
- ✓Adjust the range where the robot will run.
✗ Gets in the way
- ✗Shiny object that reflects the lamp (turns white).
- ✗Yellowish or flickering light that changes the hue.
- ✗Calibrate during the day and test at night.
Why learn this
A color mask is the lightest form of vision: it runs at dozens of frames per second on the Pi, with no AI, and is enough for a robot to follow a ball or a colored line. It’s the first version of project 3.6, before YOLO26 is introduced.
Key concepts
🎯 Object contours and center
The mask says “there’s red here,” but the robot needs two numbers: where the object how close. OpenCV finds the contours (the edges of each white blob), you keep the largest one, calculate its center and area. The center becomes the error horizontal position (how far the object is to the left or right of the image center), and the area indicates distance: the larger it is, the closer the object is.
How to read this map: The rectangle is the mask; the dashed line is the image center. Of the two blobs, only the largest counts (the small one is noise). The cyan arrow is the error: positive when the object is on the right, negative on the left, and zero when it’s centered. This is the number the PID in project 3.6 will drive to zero.
Goal: run continuously in a loop, print the center, error, and area of the largest red object, and indicate which way the robot should turn.
# Finds the largest red object and calculates its center, error, and area
import time
import cv2, numpy as np
from picamera2 import Picamera2
LARGURA, ALTURA = 640, 480
AREA_MIN = 800 # smaller than this is noise
FAIXAS = [((0, 120, 70), (10, 255, 255)), ((170, 120, 70), (179, 255, 255))]
nucleo = np.ones((5, 5), np.uint8)
def achar_objeto(quadro):
hsv = cv2.cvtColor(quadro, cv2.COLOR_BGR2HSV)
mascara = np.zeros(hsv.shape[:2], np.uint8)
for baixo, alto in FAIXAS: # combines the ranges (OR)
mascara |= cv2.inRange(hsv, np.array(baixo), np.array(alto))
mascara = cv2.morphologyEx(mascara, cv2.MORPH_OPEN, nucleo)
contornos, _ = cv2.findContours(mascara, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
if not contornos:
return None
maior = max(contornos, key=cv2.contourArea)
area = cv2.contourArea(maior)
if area < AREA_MIN:
return None
m = cv2.moments(maior)
cx, cy = int(m["m10"] / m["m00"]), int(m["m01"] / m["m00"])
return cx, cy, area
picam2 = Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"size": (LARGURA, ALTURA), "format": "RGB888"}))
picam2.start(); time.sleep(1)
print("Show a red object. Ctrl+C to exit.")
try:
while True:
achado = achar_objeto(picam2.capture_array())
if achado is None:
print("no object → stop and search")
else:
cx, cy, area = achado
erro = cx - LARGURA // 2
lado = "centro" if abs(erro) < 40 else ("direita" if erro > 0 else "esquerda")
print(f"cx={cx:3d} cy={cy:3d} error={erro:+4d} area={area:7.0f} → {lado}")
time.sleep(0.1)
except KeyboardInterrupt:
picam2.stop()How to check
- ✓Move the object to the right of the camera, and the error becomes positive and “→ right” appears.
- ✓Move the object closer and the area grows substantially; move it farther away and the area gets smaller.
- ✗Right and left reversed? The camera is upside down or mirrored: invert the error’s sign or rotate the camera.
💡 Area is not distance in meters
Area is useful for comparison (“closer,” “too far”), not for measurement. Note the area with the object at 30 cm and 1 m: those two numbers become the “reached” and “move closer” thresholds in the project.
Why learn this
Center and area are the bridge between vision and movement. YOLO26 in the next module returns a box instead of a contour, but you’ll calculate the exact same error and “size” from it.
Key concepts
⏱️ Measure FPS and reduce resolution
A robot that sees at 3 frames per second reacts late: moving at half a meter per second, it travels more than 15 cm between images. The number that matters is the FPS of the entire loop (capture + process + decide), not the camera’s. The biggest lever is resolution: half the width and half the height means one quarter of the pixels, and almost all the processing workload drops accordingly.
🌱 New here?
FPS (frames per second) counts how many complete cycles your loop runs in one second. Latency is the delay between something happening in front of the camera and the robot reacting. They go hand in hand: low FPS almost always means a delayed reaction.
How to read this map: Each bar represents the work for one frame. Going from Full HD to 640 × 480 cuts the workload by almost seven times. To track a colored ball, even 320 × 240 works; for YOLO26, 640 × 480 is the natural size.
Goal: measure the real FPS of the “capture + HSV + mask” loop at three resolutions and choose yours based on the numbers.
# Measures FPS for the complete loop at three resolutions (5 s each)
import time
import cv2
from picamera2 import Picamera2
picam2 = Picamera2()
for largura, altura in [(1280, 720), (640, 480), (320, 240)]:
cfg = picam2.create_video_configuration(main={"size": (largura, altura), "format": "RGB888"})
picam2.configure(cfg)
picam2.start(); time.sleep(1)
n, t0 = 0, time.perf_counter()
while time.perf_counter() - t0 < 5:
quadro = picam2.capture_array()
hsv = cv2.cvtColor(quadro, cv2.COLOR_BGR2HSV) # typical workload
mascara = cv2.inRange(hsv, (0, 120, 70), (10, 255, 255))
n += 1
fps = n / (time.perf_counter() - t0)
print(f"{largura}x{altura}: {fps:5.1f} FPS ({1000 / fps:5.1f} ms per cycle)")
picam2.stop()How to check
- ✓Three lines appear, and FPS rises as the resolution drops (the camera’s own frame rate is the ceiling).
- ✓Write down the numbers: they’re your reference for comparison with YOLO26 in 3.3.
- ✗FPS dropping over time? Run saude.py from 3.1: it’s probably heat.
✓ Improves FPS
- ✓Ask the camera for the final resolution (without resizing afterward).
- ✓Process only the part of the image you need (for example, the bottom half).
- ✓Print less: print printing every frame also has a cost.
✗ Reduces FPS
- ✗Save a JPEG for every frame to the card.
- ✗Open the desktop with a video window while the robot is running.
- ✗Pi without a cooler: starts fast, then slows down.
Why learn this
Without measuring, every optimization is a guess. With the FPS in hand, you can use data to choose between “fast color” and “slower AI,” and know how far the robot can travel between decisions without losing the object.
Key concepts
🌐 Stream video to the browser
The Pi on the robot has no screen, and video windows over SSH don’t work. The solution is to make the Pi a small web server: it compresses each frame into a JPEG and sends the sequence over the network in the MJPEGformat, which any browser can display as video. This lets you see exactly what the robot sees, with your code’s markings already applied, on your phone or laptop.
🌱 New here?
MJPEG is “JPEG in motion”: one photo after another over a connection that stays open. It’s simple and has low latency, but uses more network bandwidth than properly compressed video. The port (here, 8000) is the number that distinguishes services on the same computer; that’s why the address ends in :8000.
How to read this map: Inside the Pi (dashed outline), two tasks run at the same time: one only captures and prepares the latest JPEG; the other only delivers it. Keeping them separate prevents a slow browser from delaying the robot’s vision.
Goal: View the robot’s video in a browser, with the center of the red object marked. Uses only Python’s standard library, with nothing to install.
# Camera video in the browser (MJPEG) with the object center marked
import threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import cv2, numpy as np
from picamera2 import Picamera2
picam2 = Picamera2()
picam2.configure(picam2.create_video_configuration(main={"size": (640, 480), "format": "RGB888"}))
picam2.start()
ultimo_jpeg, trava = None, threading.Lock()
def capturar():
global ultimo_jpeg
while True:
quadro = picam2.capture_array()
hsv = cv2.cvtColor(quadro, cv2.COLOR_BGR2HSV)
mascara = cv2.inRange(hsv, (0, 120, 70), (10, 255, 255)) | cv2.inRange(hsv, (170, 120, 70), (179, 255, 255))
contornos, _ = cv2.findContours(mascara, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
if contornos:
maior = max(contornos, key=cv2.contourArea)
m = cv2.moments(maior)
if m["m00"] > 800:
cx, cy = int(m["m10"] / m["m00"]), int(m["m01"] / m["m00"])
cv2.drawContours(quadro, [maior], -1, (255, 0, 255), 2)
cv2.circle(quadro, (cx, cy), 6, (255, 255, 0), -1)
cv2.line(quadro, (320, 0), (320, 480), (200, 200, 200), 1) # center of the image
ok, buf = cv2.imencode(".jpg", quadro, [cv2.IMWRITE_JPEG_QUALITY, 70])
if ok:
with trava:
ultimo_jpeg = buf.tobytes()
PAGINA = b'<html><body style="margin:0;background:#111"><img src="/video" style="width:100%"></body></html>'
class Servidor(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/":
self.send_response(200); self.send_header("Content-Type", "text/html"); self.end_headers()
self.wfile.write(PAGINA)
elif self.path == "/video":
self.send_response(200)
self.send_header("Content-Type", "multipart/x-mixed-replace; boundary=QUADRO")
self.end_headers()
try:
while True:
with trava:
jpg = ultimo_jpeg
if jpg:
self.wfile.write(b"--QUADRO\r\nContent-Type: image/jpeg\r\n\r\n" + jpg + b"\r\n")
time.sleep(0.05) # up to ~20 frames per second
except (BrokenPipeError, ConnectionResetError):
pass # the browser closed the tab
else:
self.send_error(404)
def log_message(self, *args):
pass # silence the log for each request
threading.Thread(target=capturar, daemon=True).start()
print("Open http://robo.local:8000 in your browser. Ctrl+C to exit.")
ThreadingHTTPServer(("0.0.0.0", 8000), Servidor).serve_forever()How to check
- ✓On your phone (on the same Wi-Fi), http://robo.local:8000 shows the video with a vertical line down the middle.
- ✓A red object gets a magenta outline and a cyan dot at its center.
- ✗Can’t open robo.local on your phone? Use the Pi’s IP (hostname -I) instead of the name.
💡 Local network only
This server has no password: anyone on the same network can see the camera. Use it at home or in the lab, never exposed to the internet. On a school network, check with the teacher before turning it on.
Why learn this
Seeing what the robot sees is the most powerful debugging tool in this learning path. When it turns the wrong way in project 3.6, you can look at the video and tell in seconds whether the problem is vision (it marked the wrong object) or control (it marked the right one but turned the wrong way).
Key concepts
✅ Module summary
To go further
- Raspberry Pi camera software — rpicam-apps and picamera2
- Raspberry Pi cameras — models, cables, and installation
- OpenCV: color spaces — HSV and inRange step by step
- OpenCV: contours — moments, area, and bounding boxes
- OpenCV documentation — complete reference
Next module
3.3 · Detect objects with YOLO26: from “find the red thing” to “find the person, cat, or bottle,” running on the Pi 5.