🧠 AI inside the microcontroller

An AI model can run in three places: on a server on the internet, on a computer inside the robot (the Pi 5 from module 3.3), or on the microcontroller itself which already reads the sensors. The closer it is to the sensor, the lower the latency and power use, and no data leaves the robot. The trade-off is size: the ESP32-S3 can fit models ranging from tens to hundreds of kilobytes, which answer simple questions like "is this movement a wave?" or "is there a person in front?".

How to read this map: From left to right, the model gets smaller and closer to the sensor. The highlighted box is the focus of this module. The times are orders of magnitude, not measurements from your board: always measure on your setup.

Why learn this

Knowing where each model should run helps you avoid two costly mistakes: adding a Pi 5 to a project that an ESP32-S3 could handle on battery power, or trying to fit on the microcontroller a task that only a computer can handle. In the complete robot (modules 3.5 and 3.6), both work together: the Pi sees the world and the ESP32 reacts.

Key concepts

TinyML is the term for machine learning that runs on microcontrollers. The ESP32-S3 has about 512 KB of internal RAM and a few megabytes of flash; many boards add 2 to 8 MB of external PSRAM. A model trained on a PC stores each weight as float32 (4 bytes). Quantization converts each weight to int8 (1 byte, from −128 to 127): the model becomes 4 times smaller and integer calculations run much faster, with only a small drop in accuracy when done well.

How to read this map: each weight goes from taking up four boxes (bytes) to taking up one. The scale and zero point are stored in the model so it can be converted back when needed. The example numbers are illustrative.

Why learn this

When a TinyML project fails, it’s almost always because of memory: the model doesn’t fit in flash, or the arena doesn’t fit in RAM. Understanding that int8 is the standard format and knowing the approximate size of each task helps you choose the board and model before spending an afternoon compiling.

Key concepts

O Edge Impulse is a web platform with a free plan for personal projects. You send in sensor data, label it, choose processing and neural network blocks with a few clicks, train in the cloud, and download the result as an Arduino library in .zip format, already quantized. You don’t write the network; you only write the sketch that reads the sensor and calls the classifier.

How to read this map: the five boxes are the Edge Impulse project tabs, in the order you move through them. The cyan loop shows what happens in practice: the first model almost always confuses two classes, and the solution is to collect more data, not change the network.

Step by step

Why learn this

It’s the shortest path from “I have a sensor” to “I have a classifier running on the board.” In the classroom, students collect their own data and immediately see the effect of poor data or imbalanced classes—the most important lesson in any AI project.

Key concepts

Behind many TinyML projects is TensorFlow Lite Micro (TFLM), Google’s inference engine for microcontrollers. In 2024, TensorFlow Lite was renamed LiteRT; you’ll see both names. Espressif maintains the esp-tflite-micro component for ESP-IDF, with ready-to-use examples (hello_world, micro_speech, person_detection) and ESP-NN, which uses the S3’s vector instructions to speed up network layers.

How to read this map: on the left is what you prepare once (the model and the list of layers). In the center, the interpreter performs calculations in a fixed area of RAM. On the right is the cycle for each inference: write the input, call Invoke(), read the output. The dashed line below indicates ESP-NN acceleration, which kicks in automatically on the S3.

The five steps in every TFLM program

Goal: compile and flash the official example hello_world (a network that learns the sine curve) on the ESP32-S3. Requires ESP-IDF 5.x to be installed and the ESP-IDF terminal to be open.

# 1. baixar o componente com os exemplos
git clone https://github.com/espressif/esp-tflite-micro.git
cd esp-tflite-micro/examples/hello_world

# 2. escolher o chip e compilar
idf.py set-target esp32s3
idf.py build

# 3. gravar e abrir o monitor (troque a porta pela sua)
idf.py -p /dev/ttyACM0 flash monitor
# no Windows a porta é algo como COM5; para sair do monitor: Ctrl+]

Why learn this

Edge Impulse handles most cases, but when you train your own network with TensorFlow or want to control every byte of memory, TFLM is the tool to use. Understanding how the interpreter works also helps explain error messages from any TinyML library, including Edge Impulse.

Key concepts

Use the MPU-6050 from module 2.2 connected to the ESP32-S3 and teach it three classes: stopped, shake e circle. The model does not look at a single reading, but at a window: for example, 2 seconds of acceleration on all three axes, read 100 times per second. You need two sketches: one to collect data and another to classify it.

How to read this map: The three lines are the accelerometer axes. The rectangle is the window the model sees all at once. On the right is the output: a score from 0 to 1 for each class, adding up to 1. Accept the result only if the highest score exceeds a threshold (0.8 in the sketch).

Goal (data collection): print the acceleration in m/s², 100 times per second, in the format that edge-impulse-data-forwarder understands (comma-separated values, one line per reading).

// Collect data from the MPU-6050 for Edge Impulse (ESP32-S3, esp32 3.x package)
// Output: "ax,ay,az" in m/s², 100 lines per second, 115200 baud.
#include <Wire.h>

const int PINO_SDA = 8;           // replace with your board's I2C pins
const int PINO_SCL = 9;
const uint8_t MPU = 0x68;        // MPU-6050 address (AD0 to GND)
const unsigned long PERIODO_US = 10000; // 10 ms = 100 Hz
unsigned long proxima = 0;

void escreveRegistro(uint8_t reg, uint8_t valor) {
  Wire.beginTransmission(MPU); Wire.write(reg); Wire.write(valor); Wire.endTransmission();
}

void setup() {
  Serial.begin(115200);
  Wire.begin(PINO_SDA, PINO_SCL);
  Wire.setClock(400000);
  escreveRegistro(0x6B, 0x00);      // wake up the sensor
  escreveRegistro(0x1C, 0x00);      // ±2 g range: 16384 per g
  proxima = micros();
}

void loop() {
  if ((long)(micros() - proxima) < 0) return;  // wait for the next 10 ms tick
  proxima += PERIODO_US;
  Wire.beginTransmission(MPU); Wire.write(0x3B); Wire.endTransmission(false);
  Wire.requestFrom(MPU, (uint8_t)6);
  if (Wire.available() < 6) return;
  int16_t bruto[3];
  for (int i = 0; i < 3; i++) { uint8_t alto = Wire.read(), baixo = Wire.read(); bruto[i] = (int16_t)((alto << 8) | baixo); }
  const float G = 9.80665f / 16384.0f;  // counts → m/s²
  Serial.printf("%.3f,%.3f,%.3f\n", bruto[0] * G, bruto[1] * G, bruto[2] * G);
}

Goal (classification): with the exported library installed, fill the window with readings at the same rate used for training, call the classifier, and turn on the LED when the "shake" gesture scores above 0.8. Replace the name of #include with your project's header and the labels with the ones you created.

// Classify gestures with the library exported by Edge Impulse (ESP32-S3)
#include <Wire.h>
#include <seu_projeto_inferencing.h>  // replace with the generated header: <seu_projeto>_inferencing.h

const int PINO_SDA = 8, PINO_SCL = 9, PINO_LED = 2;
const uint8_t MPU = 0x68;
const float LIMITE = 0.80f;
static float janela[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE];  // 200 readings × 3 axes

void escreveRegistro(uint8_t reg, uint8_t valor) {
  Wire.beginTransmission(MPU); Wire.write(reg); Wire.write(valor); Wire.endTransmission();
}

void leAceleracao(float *a) {
  Wire.beginTransmission(MPU); Wire.write(0x3B); Wire.endTransmission(false);
  Wire.requestFrom(MPU, (uint8_t)6);
  const float G = 9.80665f / 16384.0f;
  for (int i = 0; i < 3; i++) { uint8_t alto = Wire.read(), baixo = Wire.read(); a[i] = (int16_t)((alto << 8) | baixo) * G; }
}

void setup() {
  Serial.begin(115200);
  pinMode(PINO_LED, OUTPUT);
  Wire.begin(PINO_SDA, PINO_SCL); Wire.setClock(400000);
  escreveRegistro(0x6B, 0x00); escreveRegistro(0x1C, 0x00);
  Serial.printf("Model ready: %d classes, window of %d values\n", EI_CLASSIFIER_LABEL_COUNT, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE);
}

void loop() {
  // 1) fill the window at the same rate used for training
  for (size_t i = 0; i < EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE; i += EI_CLASSIFIER_RAW_SAMPLES_PER_FRAME) {
    unsigned long t0 = micros();
    leAceleracao(&janela[i]);
    while (micros() - t0 < (unsigned long)(EI_CLASSIFIER_INTERVAL_MS * 1000)) {}
  }
  // 2) run the classifier
  signal_t sinal;
  numpy::signal_from_buffer(janela, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &sinal);
  ei_impulse_result_t resultado = { 0 };
  if (run_classifier(&sinal, &resultado, false) != EI_IMPULSE_OK) { Serial.println("classifier error"); return; }
  // 3) show the scores and decide
  bool chacoalhou = false;
  for (size_t i = 0; i < EI_CLASSIFIER_LABEL_COUNT; i++) {
    Serial.printf("%s: %.2f  ", resultado.classification[i].label, resultado.classification[i].value);
    if (strcmp(resultado.classification[i].label, "chacoalhar") == 0 && resultado.classification[i].value > LIMITE) chacoalhou = true;
  }
  Serial.printf("| %d ms\n", resultado.timing.dsp + resultado.timing.classification);
  digitalWrite(PINO_LED, chacoalhou ? HIGH : LOW);
}

Why learn this

The same approach works for detecting when the robot has tipped over, when a wheel is wobbling, or when someone has picked up the robot. These reactions need to happen locally and quickly, without waiting for the Pi.

Key concepts

Boards such as the XIAO ESP32-S3 Sense come with a microphone and camera. With them, you can run the two classic esp-tflite-micro examples: the micro_speech, which recognizes the English words "yes" and "no", and the person_detection, which gives an "is there a person" score for a 96×96-pixel grayscale image. For Portuguese words, train your keyword model in Edge Impulse using recordings from your class.

How to read this map: in both workflows, the microcontroller first shrinks the input (sound becomes a frequency map; the image becomes 96×96 grayscale) and only then calls the model. This preprocessing is half the work and must match what was used during training.

Goal: flash the official person detector to an ESP32-S3 board with a camera and PSRAM. Each board’s camera pins are listed in the example’s configuration file; check its README to see if your board is already listed.

cd esp-tflite-micro/examples/person_detection
idf.py set-target esp32s3
idf.py menuconfig      # check the board/camera and whether PSRAM is enabled
idf.py build
idf.py -p /dev/ttyACM0 flash monitor

Why learn this

A spoken “stop” that the ESP32 understands on its own is a safety lock that doesn’t depend on Wi-Fi or the Pi. And an inexpensive person detector can wake the Pi only when someone is in front of it, saving battery. In module 3.5, you’ll connect these two worlds with a serial wire.

Key concepts