Detailed content
📏 Measuring: tokens/s, latency, and memory
You can’t improve what you don’t measure. Before buying hardware or blaming the model, capture three numbers: generation speed (tokens/s), time to the first word (latency), and how much memory is being used—and by what (GPU or CPU).
🆕 New here?
- tokens/s = how many “word pieces” the model generates per second. Above ~15 tok/s, reading flows like a chat; below ~5, it gets annoying.
- latency = the wait time before the response starts coming out (loading the model + processing your prompt). Longer context increases latency.
- OOM = out of memory: memory (VRAM/RAM) ran out and the model doesn’t fit. It’s the number one error in the local lab—we’ll return to it in topic 5.
⌨️ Practical example: measure your lab
Objective: see what's loaded, where it runs (GPU vs. CPU), and its actual generation speed. Paste this into the terminal.
How to check: at the end of generation, check eval rate — is your tokens/s. In ollama ps, the column PROCESSOR shows the split: 100% GPU is ideal; if CPU shows up, part of the model has “spilled over” into RAM and slowed down.
📊 What “good enough” means
- Interactive chat: > 15 tok/s and latency < 2 s feel instantaneous to a human reader.
- 24/7 agent / batch: speed barely matters — it works on its own. What matters here is fitting in memory and avoiding OOM.
- Voice / real time: requires much more — low latency becomes the bottleneck, not average tokens/s.
🔀 Multi-GPU and multi-machine
When one machine can't handle it, you don't need a bigger machine—you need more machines communicating. The recipe combines two pieces you've already set up in the course: the private network from Tailscale (2.4) and the orchestrator that routes tasks (4.1).
Multi-GPU on the same machine
Two GPUs in the same PC: Ollama automatically splits the model layers between them. The combined VRAM of both cards can fit a larger model — great for 70B on two 24 GB cards.
Routing by machine
The giant model lives on the unified-memory Mac (slow, but it fits); the fast model runs on the NVIDIA GPU. The orchestrator sends each task to the right endpoint — a recap straight from 4.1.
Tailscale is the glue
Each machine gets a fixed IP 100.x on your tailnet. You can call the model from any device as if it were on the same LAN — without opening a port on the internet (recap 2.4).
⌨️ Practical example: route to another machine
Objective: expose the Ollama on the powerful machine to the tailnet and use it from a low-end laptop.
How to check: the response appears on the laptop, but the ollama ps only shows the model loaded in the machine A. The laptop became just a terminal — the remote GPU does the work.
💡 Practical tip
0.0.0.0 exposes Ollama to the machine's entire network. Only do this behind Tailscale — the tailnet is private and encrypted. Never expose this port to the public internet: that would leave your model (and your machine) open to anyone.
⚖️ When local is NOT worth it
Honesty wraps up the course better than hype. Local is powerful, but it isn't the answer to everything. Knowing when to use the cloud is part of being sovereign — you choose strategically, not out of dependency.
✓ Local shines
- ✓Sensitive data that cannot leave the machine
- ✓Heavy, continuous use (24/7 agents, batch)
- ✓Tasks where “good enough” gets the job done
- ✓You want independence from pricing and lockouts
✗ Time to consider the cloud
- ✗You need the absolute peak of today’s intelligence
- ✗Rare, occasional use — the hardware isn't worth it
- ✗You don’t want to maintain uptime, drivers, and security
- ✗A momentary demand spike your machine can't handle
🔀 The mature approach is hybrid
It’s not local or cloud — it's local as a foundation and cloud when it makes sense:
- •Everyday, private, and ongoing tasks run locally.
- •The rare case that requires a top-of-the-line model goes to the cloud — without sensitive data.
- •Since the local API is compatible with the OpenAI API (3.5), switching endpoints means changing a URL.
🔥 Energy and thermal costs
"Unlimited" doesn’t mean "free": keeping a GPU running 24/7 comes with an electricity bill and heat to deal with. It’s well worth it — but use it responsibly, or you’ll trade your cloud bill for an electricity bill (and equipment that’s cooking itself).
🧮 The electricity bill, on a napkin
Rule: power (kW) × hours × rate (R$/kWh). A powerful GPU draws ~350 W when it's generating (at idle, much less):
In real life, a 24/7 agent spends most of its time idle between tasks—so usage is usually a fraction of that. Even so, measure before you assume.
⚠️ Alert: heat is what kills 24/7 hardware
- ▸Ventilation: an always-on machine needs fresh air coming in and hot air going out — don’t put it in a closed cabinet.
- ▸Thermal throttling: An overheated GPU slows itself down to protect itself — your tokens/s plummets with no error at all.
- ▸Dust: after months of continuous use, it clogs the fans. Clean it periodically.
- ▸Monitor: nvidia-smi shows temperature and watts in real time; above ~83 °C, improve cooling.
🛠️ Common troubleshooting
Almost every local lab problem falls into five buckets. Instead of memorizing them, use the flow below: start with the symptom, identify the diagnostic and apply the action. The table right after this is your pocket guide.
Read from left to right: the symptom (amber) opens four diagnostics typical (cyan), and each one converges on a action for troubleshooting. The table below gives the “cause → fix” for each case.
| Problem | Likely cause | Fix |
|---|---|---|
| OOM (out of memory) | The model + context don't fit in VRAM/RAM | Use a lower quantization (Q8→Q4), reduce num_ctx, or run a smaller model |
| Slow model | Part of the model spilled over to the CPU (offload) or thermal throttling | Check ollama ps (PROCESSOR); adjust num_gpu; cool down the GPU |
| Broken driver/CUDA | Kernel/driver update desynchronized CUDA | nvidia-smi failure? Reinstall the driver and restart; reinstall the toolkit |
| Port 11434 is occupied | There’s already an Ollama (or another service) listening on it | lsof -i :11434, kill the process, or change OLLAMA_HOST |
| Tailscale offline | Session expired, service went down, or key expired | tailscale status; run sudo tailscale up again |
⌨️ Practical example: the 5 diagnostic commands
Objective: before asking for help online, run this battery of tests — it covers the five problems in the table.
How to check: Each command either responds correctly (green) or points to the culprit. With the symptom and the table row, you can solve 90% of cases on your own—without sending anything outside your machine.
🚀 Your lab from here on
You’ve reached the end—and now you have a lab of your own. What comes next is continuous improvement, not a destination. See the natural roadmap and the habits that keep everything healthy.
Keep the models up to date
Open models improve month by month. Revisit ollama pull from the new versions and test them on the your task before adopting.
Add another agent to the team
You already have the scanner (4.3), the watchdog (4.4), and the buddy (4.5). Every repetitive problem in your routine is a candidate to become a 24/7 worker.
Upgrade your hardware when a bottleneck appears
Don’t buy ahead of time. Measure (topic 1), find the real limit, and prioritize memory — the resource that matters most.
✓ Habits that keep the lab alive
- ✓Version-control configs (Modelfiles, scripts) in a repo
- ✓Log what agents do and review it
- ✓Measure before buying hardware
- ✓Keep everything on the tailnet, nothing on the public internet
✗ Pitfalls that block you
- ✗Running 24/7 without ventilation or temperature monitoring
- ✗Expose the Ollama port directly to the internet
- ✗Chasing the “perfect model” and never shipping anything
- ✗Forget the unmonitored agent with no logs or limits
💡 Practical tip
You learn local AI by building things and exchanging ideas. Communities (including the INEMA.CLUB), open-model forums, and Hugging Face cards are where you discover the next good model and avoid pitfalls others have already paid dearly for.
Self-check (optional): your model freezes with an error of OOM when loading. What’s the most straightforward first adjustment?
🏁 End of course — recap of the 4 tracks
You’ve left the theory behind and arrived at a lab that works for you. Look at the path you’ve traveled:
🤝 Stay with us
The local AI shift doesn't end here — it accelerates. To keep up, exchange ideas, show off your lab, and discover the next great model alongside others building sovereignty.
Enter the INEMA.CLUB →