🎙️ Local voice: listen and speak

A voice assistant isn't just one model: it's an assembly line with four stations. The microphone captures the sound; speech recognition turns sound into text; the brain from module 5.1 chooses and validates the action; speech synthesis turns the response into sound. Each station is a separate program that you can test and replace without changing the others.

How to read this map: The top line (teal arrows) is the path of the request, from your voice to the motor. The bottom line (dashed cyan arrows) is the return path: the brain reports what it did, and Piper speaks. The glowing box is the same one from module 5.1; voice just swaps the keyboard for a microphone.

Why learn this

Voice is the most natural interface for people who don’t program: a child, a visitor, a student at a science fair. Building the pipeline in stages teaches you to check each station and find where the delay or error is.

Key concepts

O Whisper is an open speech recognition model that understands Portuguese. Two versions work well offline: faster-whisper, in Python, which uses the CPU with 8-bit numbers (int8) or an NVIDIA GPU; and whisper.cpp, in C++, which is lightweight and works well on the Pi. This course uses faster-whisper because it works directly with the script from module 5.1. The model is downloaded from the internet the first time; after that, it works offline.

Goal: install faster-whisper and find your microphone’s number. The selected model is downloaded once during the first transcription.

source ~/robo-ia/bin/activate          # same environment as module 5.1
sudo apt install -y libportaudio2      # Linux/Pi: sounddevice needs it
pip install faster-whisper sounddevice numpy
python3 -m sounddevice                 # list microphones: note down your number

Goal: record a few seconds from the microphone and print what you said. The file also works as a module: the assistant in topic 4 imports the functions record e transcribe from here.

#!/usr/bin/env python3
"""Records audio from the microphone and transcribes it in Portuguese with faster-whisper (offline after the first download).
pip install faster-whisper sounddevice numpy"""
import time
import sounddevice as sd
from faster_whisper import WhisperModel

TAXA = 16000                # Whisper works with 16 kHz, mono
MODELO_WHISPER = "small"    # tiny, base, small, medium... larger = more accurate and slower
MICROFONE = None            # None = system default; or the number from: python3 -m sounddevice
_modelo = None

def carregar():
    global _modelo
    if _modelo is None:       # CPU int8 runs on the Pi 5 and any PC
        _modelo = WhisperModel(MODELO_WHISPER, device="cpu", compute_type="int8")
        # with an NVIDIA GPU: WhisperModel(MODELO_WHISPER, device="cuda", compute_type="float16")
    return _modelo

def gravar(segundos=4.0):
    print(f"Recording for {segundos:.0f} s... speak now.")
    audio = sd.rec(int(segundos * TAXA), samplerate=TAXA, channels=1, dtype="float32", device=MICROFONE)
    sd.wait()
    return audio.flatten()

def transcrever(audio):
    segmentos, _info = carregar().transcribe(audio, language="pt", vad_filter=True, beam_size=1)
    return " ".join(s.text.strip() for s in segmentos).strip()

if __name__ == "__main__":
    carregar()
    while True:
        input("Press Enter to speak (Ctrl+C to exit) ")
        audio = gravar()
        t0 = time.perf_counter()
        texto = transcrever(audio)
        print(f"You said: '{texto}'  (transcription in {time.perf_counter() - t0:.1f} s)")

Why learn this

Speech recognition is the entry point: if it hears “thirty” as “three,” everything else goes wrong too. Measuring the time and testing different model sizes on your hardware will determine whether your robot responds in one second or ten.

Key concepts

The robot shouldn’t transcribe everything it hears. It needs a trigger to start paying attention. The simplest and most reliable option is push to talk: a key on the PC or a button on the Pi’s GPIO. The alternative is a wake word (“hey, robot”), detected by a tiny model that runs all the time; open projects such as openWakeWord do this, and you can train your own word (check the documentation for the project you choose).

How to read this map: The robot spends most of its time in the pulsing circle on the left. Only the trigger takes it to “listening.” Any bad result (empty text, rejected action, no model response) follows the cyan arrow back to “waiting,” without moving the motor.

Why learn this

A robot that obeys every sentence in the room is unpredictable. The trigger and proper microphone placement solve most recognition problems before you need a larger model.

Key concepts

The text produced by Whisper goes exactly where the text you typed went in module 5.1. Nothing changes in the validator: voice is just another input method. The new feature is spoken confirmation: for large movements, the robot asks "confirm?" and only proceeds if it hears "yes".

Step by step

Goal: the full assistant: press Enter to speak, transcribe, choose an action with the local model, validate, confirm by voice when the movement is large, send the command to the ESP32, and reply aloud, with the time for each step shown in the terminal.

#!/usr/bin/env python3
"""Offline voice assistant: listen (Whisper) -> choose (local model) -> validate -> ESP32 -> speak (Piper).
In the same folder: cerebro_llm.py (module 5.1), ouvir.py and falar.py (this module).
Usage: python3 assistente_voz.py [--simular]"""
import sys, time
from ouvir import carregar, gravar, transcrever
from falar import falar
from cerebro_llm import perguntar_modelo, validar, abrir_serial, enviar

FRASES = {"ANDAR": "moving {} centimeters", "GIRAR": "turning {} degrees",
          "GARRA": "gripper: {}", "PARAR": "parando"}

def em_palavras(linha):                        # "ANDAR 30" -> "moving 30 centimeters"
    partes = linha.split()
    molde = FRASES.get(partes[0], linha)
    return molde.format(*[p.lower() for p in partes[1:]]) if "{}" in molde else molde

def main():
    simular = "--simular" in sys.argv
    carregar()
    s = abrir_serial(simular)
    falar("Ready. Press Enter and speak.")
    while True:
        try:
            input("\n[Enter] to speak, Ctrl+C to exit ")
        except (KeyboardInterrupt, EOFError):
            break
        audio = gravar(4)
        t0 = time.perf_counter(); texto = transcrever(audio); t1 = time.perf_counter()
        print(f"Heard: '{texto}'")
        if not texto:
            falar("I didn't understand. Could you repeat that?"); continue
        try:
            resp = perguntar_modelo(texto)
        except Exception as e:                    # no response or unreadable JSON: nothing moves
            print("Model error:", e); falar("My brain didn't respond."); continue
        t2 = time.perf_counter()
        ok, linha, confirmar = validar(resp)
        if not ok:
            print("Recusado:", linha); falar("I can't do that."); continue
        if confirmar:
            falar(f"Confirm {em_palavras(linha)}? Say yes or no.")
            palavras = transcrever(gravar(2.5)).lower().replace(".", " ").replace(",", " ").split()
            if "sim" not in palavras:            # whole word: "assim" does not confirm
                falar("Cancelado."); continue
        print("ESP32:", enviar(s, linha))
        falar(em_palavras(linha))
        t3 = time.perf_counter()
        print(f"Times: listen {t1 - t0:.1f} s · think {t2 - t1:.1f} s · execute and speak {t3 - t2:.1f} s")
    if s:
        s.write(b"PARAR\n"); s.close()

if __name__ == "__main__":
    main()

Why learn this

Combining tested parts is how real systems are built. Since each file has a clear role, the final project in module 5.6 reuses all three without rewriting anything.

Key concepts

O Piper is an open source speech synthesizer, fast enough for Raspberry Pi. Each voice is an .onnx file (the model) with an .onnx.json file alongside it (the configuration). Brazilian Portuguese voices include pt_BR-faber-medium; the full list is in the voice repository. The project moved from its original home (rhasspy/piper) and continues as piper1-gpl, from the Open Home Foundation; the piper-tts pip package follows this change, and the synthesis function has been renamed between versions. The script below works with both.

How to read this map: from left to right, the sentence is broken down into the sounds of the language (phonemes), which the voice model turns into audio. Changing the voice means changing just the file in the glowing box; the rest of the process stays the same.

Goal: install Piper and download a Portuguese voice. At the end, the files pt_BR-faber-medium.onnx e .onnx.json should be in the folder.

pip install piper-tts
python3 -m piper.download_voices pt_BR-faber-medium   # downloads .onnx and .onnx.json to the current folder
# old package version? download both voice files from huggingface.co/rhasspy/piper-voices

Goal: a function falar(texto) that synthesizes speech in memory and plays it through the default speaker, without saving a file to disk.

#!/usr/bin/env python3
"""Speaks text in Portuguese with Piper, offline. pip install piper-tts sounddevice numpy
Usage: python3 falar.py "Hello, I'm the robot." """
import io, sys, wave
import numpy as np
import sounddevice as sd
from piper import PiperVoice

VOZ = "pt_BR-faber-medium.onnx"    # the .onnx.json must be alongside it
_voz = None

def falar(texto):
    global _voz
    if _voz is None:
        _voz = PiperVoice.load(VOZ)
    buf = io.BytesIO()
    with wave.open(buf, "wb") as wav:
        if hasattr(_voz, "synthesize_wav"):   # newer versions of piper-tts
            _voz.synthesize_wav(texto, wav)
        else:                                 # older versions
            _voz.synthesize(texto, wav)
    buf.seek(0)
    with wave.open(buf, "rb") as wav:
        taxa = wav.getframerate()
        dados = np.frombuffer(wav.readframes(wav.getnframes()), dtype=np.int16)
    sd.play(dados, taxa)
    sd.wait()                                 # wait until it finishes: the robot can't hear itself speaking

if __name__ == "__main__":
    falar(" ".join(sys.argv[1:]) or "Hello, I am the course robot. I'm ready.")

How to check: python3 falar.py "Turning ninety degrees" plays the Portuguese sentence in less than a second on a PC (a little longer on the Pi). If you get "No such file", the name in VOICE doesn't match the downloaded file; if there's no sound, check the output device with python3 -m sounddevice.

Why learn this

A robot that tells you what it's about to do is safer and easier to debug: you hear "turning ninety degrees" before it moves and have time to hit stop if that wasn't what you wanted.

Key concepts

Latency is the time between when you stop speaking and when the robot responds. It's the sum of the steps, and one of them almost always dominates. That's why the assistant prints the times separately: optimize the slowest step, not the one that just seems slowest. The figure shows a typical setup on a PC without a GPU; your hardware will have different numbers, so measure them.

How to read this map: each bar is a step; its length shows the time. The dashed bar doesn't depend on the computer; it depends on how much audio you record. The pulsing bar is usually the language model: that's where a smaller model or a GPU makes the biggest difference.

Privacy is the other half of the choice. With a cloud voice service, the audio from your room travels to a third-party server. In a classroom, with children’s voices, this requires permission and care with data protection laws. In the local pipeline, sound is processed and discarded inside the robot.

How to read this map: On the left, the short arrow stays inside the teal frame: the audio never leaves the device. On the right, the dashed red arrow crosses the internet. The cloud may be more accurate or faster; the choice depends on who is speaking near the robot and what you can send outside.

Why learn this

Measuring latency turns “it’s slow” into “the language model takes this long,” which has a solution. Thinking about privacy from the start also prevents you from having to redo the project when it goes to a school, a fair, or someone’s home.

Key concepts