Lesson 7 · 18 min
The same CLI that renders also speaks and listens, all on your machine: it generates the narration with a local voice, transcribes word by word with the timing of each one, and hands that to the composition. In this lesson you build the technical side of sound: narration, a rehearsal soundtrack, and the karaoke caption that lights up each word the moment it is spoken.
Start the lesson
<audio> with id, data-start, and data-volume in the compositionGoal: generate local narration, get the timing of each word, and render a video with audio and karaoke captions.
Click each step. Nothing leaves your machine.
Voice, soundtrack, and mixing as a creative choice: course 2 (Production). Here, only how to put sound in the code.
One sentence, the local voice in Portuguese, and the transcription with timing.
K=~/projetos/formacao-cinema-ia-motion-v72/kit
export HYPERFRAMES_NO_TELEMETRY=1 NO_COLOR=1
E=${TMPDIR:-/tmp}/ensaio-motion/a7; rm -rf "$E"; mkdir -p "$E"
cp -r "$K/exemplos/a7-karaoke" "$E/karaoke"; cd "$E/karaoke"
TEXTO="A voz sai do seu computador, e cada palavra sabe a hora de aparecer."
npx -y hyperframes@0.8.176 tts "$TEXTO" --voice pf_dora --lang pt-br -o narracao.wav > tts.log 2>&1; echo "tts: $?"
ffprobe -v error -show_entries format=duration -of csv=p=0 narracao.wav
npx -y hyperframes@0.8.176 transcribe narracao.wav -m small -l pt --json > transcricao.log 2>/dev/null; echo "transcribe: $?"
python3 -c "import json; print('palavras:', len(json.load(open('transcript.json'))))"tts: 0 4.074667 transcribe: 0 palavras: 14
The second line is the narration length in seconds. The word count checks the transcript, not the text: check the words by opening transcript.json.
The default transcribe model in 0.8.176 is small.en, English only. For Portuguese, pass -m small -l pt: the multilingual small doesn't appear in the --help list (tiny.en, base.en, small.en, medium.en, large-v3), but it worked on this machine (screenshot alongside). And with --engine auto (default) the CLI uses Parakeet if it's installed and covers the language; without it, whisper.cpp.
A title, the narration starting at 0.5 s, a rehearsal soundtrack at 25%, and captions generated from the transcript.
<!doctype html>
<html lang="pt-BR">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=1920, height=1080" />
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
<script src="legendas.js"></script>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
html, body { width: 1920px; height: 1080px; overflow: hidden; background: #0c0c10; }
#root { width: 100%; height: 100%; position: relative; font-family: ui-sans-serif, system-ui, sans-serif; }
#luz { position: absolute; inset: 0; background: radial-gradient(ellipse at 50% 40%, #1c1c22 0%, #0c0c10 70%); }
#titulo { position: absolute; left: 0; right: 0; top: 300px; text-align: center; color: #f4f4f5; font-size: 110px; font-weight: 700; line-height: 1.1; }
.legenda { position: absolute; left: 0; right: 0; top: 820px; text-align: center; font-size: 70px; font-weight: 700; line-height: 1.2; color: #a1a1aa; opacity: 0; }
.legenda span { margin: 0 12px; }
</style>
</head>
<body>
<div id="root" data-composition-id="main" data-start="0" data-duration="8" data-width="1920" data-height="1080">
<div id="luz"></div>
<h1 id="titulo">Som no código</h1>
<audio id="narracao" src="narracao.wav" data-start="0.5" data-track-index="1"></audio>
<audio id="trilha" src="trilha.wav" data-start="0" data-duration="8" data-volume="0.25" data-track-index="2"></audio>
</div>
<script>
// legendas.js (generated by karaoke.py) brings each caption with the timing of each word
const root = document.getElementById("root");
const tl = gsap.timeline({ paused: true });
tl.fromTo("#titulo", { y: 30, opacity: 0 }, { y: 0, opacity: 1, duration: 0.6, ease: "power3.out" }, 0.1);
window.LEGENDAS.forEach((linha, i) => {
const el = document.createElement("p");
el.className = "legenda";
el.id = "legenda-" + i;
linha.palavras.forEach((p, j) => {
const s = document.createElement("span");
s.id = `palavra-${i}-${j}`;
s.textContent = p.t;
el.appendChild(s);
tl.set(s, { color: "#f4f4f5" }, p.inicio); // the word lights up when it is spoken
});
root.appendChild(el);
tl.set(el, { opacity: 1 }, linha.inicio);
tl.set(el, { opacity: 0 }, linha.fim);
});
window.__timelines["main"] = tl;
</script>
</body>
</html>#!/usr/bin/env python3
"""karaoke.py — turns transcript.json (word + timing) into legendas.js for the composition.
Usage: python3 karaoke.py transcript.json --inicio 0.5 --por-linha 3 > legendas.js
--inicio seconds at which the narration starts in the video (the <audio> data-start)
--por-linha words per caption (default 3)
Prints to stderr: "palavras: N · legendas: M".
"""
import argparse
import json
import sys
def main():
a = argparse.ArgumentParser()
a.add_argument("transcript")
a.add_argument("--inicio", type=float, default=0.0)
a.add_argument("--por-linha", type=int, default=3)
o = a.parse_args()
palavras = json.load(open(o.transcript, encoding="utf-8"))
linhas = []
for i in range(0, len(palavras), o.por_linha):
grupo = palavras[i:i + o.por_linha]
linhas.append({
"inicio": round(grupo[0]["start"] + o.inicio, 3),
"fim": round(grupo[-1]["end"] + o.inicio, 3),
"palavras": [{"t": p["text"], "inicio": round(p["start"] + o.inicio, 3)} for p in grupo],
})
# each caption leaves 0.15 s after the last word, but never after the next one enters
for atual, prox in zip(linhas, linhas[1:] + [None]):
atual["fim"] = round(min(atual["fim"] + 0.15, prox["inicio"]) if prox else atual["fim"] + 0.15, 3)
print("window.LEGENDAS = " + json.dumps(linhas, ensure_ascii=False) + ";")
print(f"palavras: {len(palavras)} · legendas: {len(linhas)}", file=sys.stderr)
if __name__ == "__main__":
main()K=~/projetos/formacao-cinema-ia-motion-v72/kit
export HYPERFRAMES_NO_TELEMETRY=1 NO_COLOR=1
cd ${TMPDIR:-/tmp}/ensaio-motion/a7/karaoke
# rehearsal soundtrack: a soft 8 s chord generated in ffmpeg (your real soundtrack goes in its place)
ffmpeg -v error -y -f lavfi -i "aevalsrc=0.10*sin(2*PI*220*t)+0.07*sin(2*PI*277.18*t)+0.06*sin(2*PI*329.63*t):s=48000:d=8" \
-af "afade=t=in:d=1,afade=t=out:st=6.5:d=1.5" trilha.wav
python3 "$K/karaoke.py" transcript.json --inicio 0.5 --por-linha 3 > legendas.js
npx -y hyperframes@0.8.176 check 2>&1 | grep -E "Check passed|✗"
npx -y hyperframes@0.8.176 render --quality draft --workers 2 --quiet --output karaoke.mp4 > render.log 2>&1
echo "render: $?"
python3 "$K/conferir.py" karaoke.mp4 8 1920x1080 --audio; echo "saída: $?"palavras: 14 · legendas: 5 ◇ Check passed render: 0 OK: 8,0 s · 1920×1080 · 30 fps · com áudio saída: 0
The checker with --audio rejects a video with no sound track. The soundtrack is just a chord generated by FFmpeg for the rehearsal; yours goes in place of trilha.wav.
In the first version, each caption stayed 0.15 s after the last word even when the next one had already come in: two captions overlapped in the frame. karaoke.py now closes each caption at most when the next one opens. It only showed up by looking at the frames.
Voice, karaoke and your sentence.
Run the "tts, duration and transcription" block. The first use downloads the voice and the model.
Done when: tts: 0, one line with the duration (a number greater than 0), transcribe: 0 and palavras: followed by a number greater than 0.
Run the "soundtrack, captions, render and check" block and watch karaoke.mp4.
Done when: ◇ Check passed, render: 0 and OK: 8,0 s · 1920×1080 · 30 fps · com áudio with saída: 0.
Swap TEXTO for a sentence of your own, up to 5 s, run both blocks again (the second after the first), and check the video.
Done when: python3 ~/projetos/formacao-cinema-ia-motion-v72/kit/conferir.py karaoke.mp4 8 1920x1080 --audio; echo "saída: $?" → a line ending in com áudio and saída: 0.
The explanation appears after you answer.
Question 1 of 4
The render came out silent, but the <audio> is in the composition. What should you check first?
Question 2 of 4
Why do the captions come from a legendas.js saved in the folder and not from a fetch at render time?
Question 3 of 4
The narration starts at 0.5 s in the video. Where does that delay go?
Question 4 of 4
Which transcribe model should you use for Portuguese narration?
Pick one option. The explanation appears after your attempt.
<audio> with id and data-start; data-volume for the soundtrack; the checker charges for the track.