O Guia de IA Local
Engenharia Guia
Execute IA de nível avançado em um hardware que pertence a você.
Four ways through this: the Guide is the full reference, the Walkthrough tells the build story start to finish, Motores is how to choose an inference engine.
New, and the most plain-English of all: a companion page of architecture blueprints, with a system diagram and an everyday-language explanation for every setup, from a single laptop to a home cluster.
Comece aqui em 60 segundos
Big AI usually runs in someone else's cloud. That costs money, can change or go down without warning, and means your data leaves your hands. This guide is about running it on your own computer instead: cheaper, private, and yours. The trade is that you set it up and look after it.
A ideia inteira em três passos:
- Run a model on your own machine with a free program (llama.cpp), on the computer you already have.
- Point your AI tool at it by changing one setting, so it uses your model instead of the cloud.
- Add more only when it hurts: speed, more users, big-codebase tricks, or safety, one piece at a time.
Everything below is the detail behind those three moves. Brand new? See the the guide or the architecture blueprints first.
A pilha completa em uma visão geral
This is the map of all the pieces and how they stack up. You do not need every layer. Most people use just the bottom few (a machine, an engine, a model) and add the higher ones only when they hit a wall.
Local AI engineering is more than installing a chat app. There is a full stack underneath, and this guide covers it top to bottom.
Por que usar IA local
The case is not "local is smarter than Claude." It is "some reasoning should never hit someone else's API," plus hard economics. The subsidy is a trap: cheap cloud coding was venture-subsidized, and the bill is coming due.
- The cloud is non-deterministic and silently degraded. A $43 experiment found output quality is "a sinusoidal wave of genius and gibberish" tied to load and time of day: aggressive quantization down to 2-bit, GPU spin-down, swapped system prompts. Local is "deterministically dumb" but predictable, and yours.
- Vendor risk is real and sudden. Outages, 5-year data-retention changes, weekly rate limits, export-control suspensions of entire models. "The outage is temporary. The weights are forever."
- The honest counter-frame. Local is about resilience and privacy, not beating the frontier on raw IQ. Run what you can locally, route only the genuinely hard, high-value, compressible problems out. Hybrid is also the only real cure for Shadow AI.
Hardware
A representative fleet runs from "sock drawer" to "space heater," all real machines under load.
| Rig | Silicon | Memory | Role |
|---|---|---|---|
| NVIDIA GB10 / ZGX Nano | Grace-Blackwell, CUDA cap 12.1 | 128GB unified | Daily-driver agentic box. Palm-sized, 33 to 74W, runs DeepSeek V4 at 1M context. |
| HP Z8 Fury G5 | Xeon + 4x RTX A6000 48GB | ~192GB VRAM / 477GB RAM | The "beast." Throughput records, training, giant MoE. ~1.8kW, A6000s at 84 to 90C. |
| HP Z2 Mini G1a | AMD Ryzen AI MAX+ PRO 395 | 128GB unified | The AMD / ROCm side. Backpack "exocortex." |
| MacBook Air / Pro | Apple Silicon M3 / M4 / M5 | 24 to 32GB | Thin SSH client; small models on Metal / MLX; WebGPU in-browser inference. |
| 2018 ThinkPad X280 | i5-8350U, CPU only | 16GB | Extreme-edge proof: Qwen3-8B at ~3.27 tok/s, ~16W on battery. |
| k8s cluster "Z14 Mecha" | 1x Z8 + 3x Z2 (NVIDIA + AMD) | 564GiB VRAM | Mixed-vendor Kubernetes, "datacenter in a closet." |
- Old hardware plus good engineering wins. "It is not the car, it is the driver." 5-year-old A6000s out-throughput SaaS APIs.
- Force more VRAM on Ryzen AI 395. Rated 96GB, but a BIOS "edge AI server" mode plus Linux GTT / TTM tweaks unlock the full 128GB pool: Qwen3-Coder-30B BF16 at 1M context, 115GB footprint, 24.4 tok/s.
- Watch thermals on Strix Halo. It hit thermal shutdown because thermald did not recognize the new CPUID; the fix was thermald --ignore-cpuid-check.
- The network is the compute. One A6000 can be shared over LAN to a ThinkPad via a virtual GPU driver, tunneling CUDA over QUIC / TLS / UDP. DGX Spark is a learning toy, not production.
Inferência e disponibilização
An inference engine is the program that actually runs the model: it loads the weights, takes your request, and streams back the answer. The model is the engine block; this is the whole car around it. vLLM is the heavy-duty one built for many users at once. llama.cpp is the go-anywhere one that runs on almost any machine.
vLLM
The production serving engine. OpenAI plus Anthropic and Ollama-compatible routes, continuous batching, chunked prefill, prefix caching. Built from source for day-zero models.
llama.cpp
The ease-of-use and edge engine. GGUF, llama-server, runs on CUDA / Metal / ROCm / Vulkan. Source edited directly for KV and quant work.
SGLang
EAGLE speculative decoding, GLM tool and reasoning parsers. A serving path for specific models.
ktransformers
Hybrid VRAM + RAM offload for huge models. Ran Kimi K2 at 1T params locally this way.
AIBrix
k8s-native control plane: LLM-aware autoscaling, distributed KV-cache routing, high-density LoRA. A capable "enterprise AI platform."
shim fake-ollama
A Rust / axum proxy that makes Copilot, Cline, and Roo think they talk to Ollama while traffic hits vLLM / llama.cpp.
The drop-in pattern that recurs constantly: point any cloud-shaped coding agent at a local endpoint.
export ANTHROPIC_BASE_URL=http://localhost:8080 # llama.cpp
export ANTHROPIC_BASE_URL=http://localhost:8000 # vLLM
# Claude Code now "just works" against your own model
When an agent speaks only one dialect, shim it. Claude Code Router (ccr) translates Claude Code into OpenAI-compatible backends; fake-ollama covers the Ollama dialect. Serious setups avoid Ollama, preferring vLLM and llama.cpp for control.
The split that decides everything is prefill versus decode. Prefill reads the prompt and is compute-bound. Decode generates one token at a time and is memory-bandwidth-bound. Short prompt with a long answer leans on bandwidth and batching; long prompt with a short answer leans on attention kernels and chunked prefill. Match the engine to that shape, your interconnect, and your quant format.
| Sua situação | Use |
|---|---|
| Notebook, edge ou hardware incomum | llama.cpp |
| Fluxos centrados no Mac | MLX / MLX-LM |
| Inferência local em uma única RTX | ExLlamaV2 |
| De 2 a 4 ou mais GPUs NVIDIA/CUDA | ExLlamaV3 |
| Inferência geral em produção | vLLM |
| Contexto longo / MoE / roteamento | SGLang |
| Máximo desempenho em NVIDIA | TensorRT-LLM |
| Orquestração de cluster | NVIDIA Dynamo |
That is the one-page version. The Engines tab has the full engine-by-engine breakdown, the real bottlenecks, hardware-strategy recipes, and how to benchmark them honestly.
The open-weight model landscape
These are the actual brains you can download and run, mostly free and open. The names change every few weeks, but the rule does not: pick the smallest one that does your job well, not the biggest one you can find.
Qwen3.x / 3.6
The hero family, 0.6B to 480B-A35B. Qwen3-Coder-Next (~80B / 3B active) is the default daily coder. "Qwen is the new Llama."
GLM-4.x / 5
Z.ai, MIT. "Opus at home." GLM-4.7-Flash is the "fast, good, and cheap" pick; topped tool-calling at a fraction of Opus cost.
DeepSeek V3.x / V4
"Soul of a Fortune-500 senior dev" for backends. V4-Flash: 284B / 13B active, 1M context, Sparse Attention ~9x cheaper.
Kimi K2.x
Moonshot, 1T / 32B active. Agent-swarm native, up to 300 parallel subagents.
MiniMax M2.x
~230B / 10B active. "Anthropic in a bottle," NVFP4-shrunk to fit a single GB10.
Gemma 4 / Nemotron
Google Apache-2.0 and NVIDIA Mamba-Transformer hybrids. Plus Ling / Ring, ERNIE, Hunyuan, LongCat.
- Efficiency is the new scaling. Bigger is not better. A 27B dense or a 3B-active MoE on the Pareto frontier beats a 1T giant for most work.
- Match the model to the language and job. DeepSeek for enterprise backends, not startup JS. Small and tiny models for narrow, repetitive agent tasks.
- Public benchmarks rarely predict fit. Token efficiency, cost, and usability on your own tasks are better selection criteria.
- MoE sparsity is the unlock. Huge total params, tiny active params per token, is what makes frontier behavior fit in consumer memory.
Judge it yourself: evals
A model topping a public leaderboard tells you almost nothing about whether it is good at your work. The fix is an eval: a small set of your own real tasks that you run every candidate model through, then keep whichever wins on those, not on the chart.
How to run a quick eval with no framework: collect 10 to 30 tasks you actually do (a real bug to fix, a function to write, a doc to summarize), each with the answer you would accept. Run every candidate model on all of them. Score pass or fail, or just A/B which output you would ship. Keep the winner on your set. That number beats every benchmark, because it is measured on the work you care about.
It cuts both ways, which is the whole point. Open models now sometimes beat the closed flagships on real-world evals (one open model recently passed a top closed model on a production web-framework eval), and sometimes they lose on yours. You only find out by testing on your own tasks. To score at scale you can use a strong model as the judge, but a human eye on 20 examples is usually enough to decide.
Compressão
Models and their memory are huge, so you shrink them. Quantization stores the numbers with less precision, like saving a photo as a smaller JPEG: slightly less exact, but it fits and still looks fine. The KV cache is the model's short-term memory of the current conversation; compressing it is what lets a giant context fit on a small graphics card.
This is where local engineering gets hard and most of the wins live. Weight formats: GGUF (Q2_K to Q8_0), AWQ 4-bit, FP8, NVFP4 / MXFP4, and BitNet 1.58-bit ternary.
APEX
MoE-aware mixed precision on stock llama.cpp. Per-expert, per-layer bits beat Q8_0 and even F16 on perplexity at half the size.
BitNet distill
Convert FP LLMs to 1.58-bit ternary: ~10x memory savings, 2.65x faster CPU inference, near-FP16 accuracy.
Custom 2-bit
A custom DeepSeek-V4 GGUF (IQ2XXS-w2Q2K) is what makes a 284B model physically fit.
TurboQuant (Google's method, implemented in vLLM's Triton path) compresses KV cache to 2.5 to 3.5 bits per token via outlier handling, a Hadamard MSE transform, Lloyd clustering codebooks, and QJL 1-bit sign projection. This is what held 4.7M tokens of KV cache resident to hit 3,426 tok/s.
vllm serve /models/target \
--tensor-parallel-size 4 --attention-backend TRITON_ATTN \
--kv-cache-dtype turboquant35 --enable-turboquant \
--turboquant-metadata-path /models/target/turboquant_kv.json
Engenharia de contexto
Context is everything the model can see at once. Like a desk, it stops working when you pile too much on it. The trick is to quit dumping the whole codebase on the desk and instead let the model fetch only the few pages it needs, exactly when it needs them.
RLM
Recursive Language Models. Load a 32M-token codebase into a Python REPL variable; the root model writes code to walk it and spawns subagents that return distilled symbols, not text walls. 15% to 40% on long-horizon tasks, zero context rot.
MemAgent
Mega-context is a memory problem, not an attention problem. Chunk-wise RL over a fixed-token memory: 3.5M-token QA at under 5% loss, scaling linearly, on a base model with as little as 8K context.
Optical compression
The "JPEG moment." Render text as images for a VLM. DeepSeek-OCR: 97% decode at 10x compression. Glyph: 10k words to ~3k visual tokens.
Sparse attention
DeepSeek Sparse Attention drops 128K prefill from ~$0.70 to ~$0.10 per Mtok.
An awk one-liner against vLLM /metrics proves it: prefix_cache_hits_total / prefix_cache_queries_total. The point: marginal cost is near zero, so the limits are pricing theater.
Context engineering handles a single conversation. For knowledge that should persist across sessions, build a private memory layer entirely on your own machine: local embeddings to index your notes and docs, plus a local re-ranker that re-sorts the top hits so the model sees the few best chunks, not the merely-similar ones. The re-ranker is the part most people skip, and it is what makes local retrieval feel sharp. None of it leaves your machine.
Local models do voice and vision too. Open multimodal models (Gemma-class) read images and run fully offline, and on-device speech models give you private dictation and transcription with no cloud. The same own-your-AI rules apply: pick a model that fits, serve it locally, keep the data on the machine.
Otimização de desempenho
Throughput is how many words per second you get out. The surprise is that the limit is usually how fast the machine can move data around in memory, not how fast it can do math. So the real wins come from smarter shuffling and scheduling, not from a bigger graphics card.
- Speculative decoding stacks. DFlash (block-diffusion drafting, ~6x speedup) plus DDTree (single-pass draft tree, tree-attention verification, lossless). Watch mean acceptance length in the logs.
- MoE expert placement is HBM-bound (limited by how fast the card's memory moves data, not by its math). For 370 TPS from 4x A6000: pin the router thread to the nearest memory, re-tile the KV cache so live experts land on the fastest memory banks, and set queue depth to ~64 to hide latency.
- Day / night daemon split. Interactive low-latency by day, async long-context bulk work by night, same hardware.
| Setup | Result |
|---|---|
| Qwen3.5-35B AWQ, 4x A6000, TurboQuant 3.5 | 3,426 tok/s at 1M context, 4.7M-token KV cache resident |
| MiniMax M2 BF16/FP8, 4x A6000, vLLM + AIBrix | 620 tok/s, 6,000 concurrent sessions at 100% success, ~1.8kW |
| GLM-4.7-Flash on a single GB10 | ~1,470 tok/s peak gen, 195 concurrent, zero queue |
| Gemma-4 31B AWQ on ONE A6000 | 129 concurrent subagents, 100% success |
| DeepSeek V4-Flash, 1M context, single GB10 | 80 to 91 tok/s per agent |
| Gemma4 2.3B in a browser tab (WebGPU, MacBook Air) | 80 tok/s, 2GB RAM per tab, fully offline |
| Qwen3-8B on a 2018 i5 CPU | 3.27 tok/s at ~16W |
Coding agents on local models
The pattern is always the same: take a cloud-shaped agent and repoint it at local weights.
Once the endpoint is yours, the interface is portable: any device that can hold a connection can act as a client, and a laptop can run the whole loop fully offline with a small local model.
Enxames e orquestração
A swarm is just many copies of the agent working at the same time, turning one worker into a whole crew. The hard part is no longer the work itself, it is handing out jobs and keeping the workers from stepping on each other.
At 10,000+ agents and a 35:1 AI-to-human ratio, there is no human UI; the backlog is the HUD.
- Swarms. Spawn, execute, report, die. Self-orchestrating swarms cut an OpenClaw deep-sweep from 3.5 hours and 19M tokens to ~10 minutes with 90% identical results.
- Isolation. Git-worktree per agent, or AgentFS (a SQLite .db holding files plus state plus tool_calls, so cp agent.db snapshot.db is time travel and you can SQL-query agent behavior).
- Legacy code is a graph, not a document corpus. A tool like dede exposes a Roslyn dependency graph as a skill so agents query blast radius instead of RAG-chunking across 40 repos.
- Headless loop. systemd timers plus flock plus Radicle issue labels as a state machine. Offline-first, human-merge-at-the-end.
Segurança e confiança zero
Once an agent can run commands, treat it like a new intern: helpful, but you would not hand it every key on day one. The setup here lets it suggest actions while a checker approves them and a sealed room contains any mistakes.
Hyperlight
CNCF, Rust. Hardware-isolated micro VMs that spawn in ~1ms with microsecond guest calls and snapshot / restore, so a hallucinated rm -rf rewinds in microseconds.
Firecracker + AgentFS
microVM isolation for agents, formally verified with Quint.
Aegis
A zero-trust install gate. Ed25519-signed execution plans, SHA-256 hash-chained audit logs, AI architecturally incapable of escalation: never gets root, never generates argv, enforced via std::process::Command with no shell.
Supply-chain hygiene
Pin every artifact to a SHA-256, disable npm lifecycle scripts, generate an SBOM, sign with cosign, gate with Trivy and semgrep. "Can your stack survive pip install?"
Self-improvement & local training
You can also make a model better on your own machine, not just use it. The cheapest trick is to let it learn from its own best answers. You do not need a data center to get a real improvement.
Self-distillation
Apple SSD: SFT on the model's own unverified outputs, no RL, verifier, teacher, or execution. Qwen3-30B went 42.4% to 55.3% pass@1 on LiveCodeBench, +30% relative.
GEPA + DSPy
Genetic-Pareto prompt and skill optimization converges in ~50 iterations vs 5,000 brute-force, auto-harvesting high-success sessions.
Tiny recursive models
TRM (7M params, recursive self-calls) hits 45% ARC-AGI-1, beating models thousands of times larger. RSA lifts Qwen3-4B +29.9 on AIME with no verifiers.
From-scratch training
nanochat end to end on 4x A6000. Rust borrow-checker RL loops (prompt, cargo check, reward) hit 71.8% compile rate in a non-root sandbox.
Close the loop
The cheapest self-improvement: give the agent a way to verify its own output (tests, a compiler, a checker), then loop. It drafts, checks, and fixes until it passes. This is how teams run swarms of self-improving agents, no training required.
Open post-training
The RL stacks behind the big open models are themselves open. One of them post-trained a frontier-class model in about two days. You can use the same tools to post-train your own model on your own data.
Casos de uso relacionados
- Reverse engineering, fully local. Ghidra plus a local agent at 80 tok/s decompiled a 6.9MB binary in 110 seconds (9,426 functions, 250+ Windows APIs, 7 auto-generated docs). "Three weeks of RE engineers becomes a Sunday morning."
- Private OCR. GLM-OCR (0.9B) beats Gemini and GPT on doc parsing while running on vLLM.
- Privacy as capability. Local Qwen3.5-35B did text deanonymization at ~90% precision, the point being your writing style is a biometric, so it should never leave your machine. The same approach reverse-engineered an air fryer's API with a local model and mitmproxy.
- Real-time / OT edge AI. CASA: 100Hz perception, 10Hz decision, sub-10ms action, with Decision Contracts (TTL, multi-model consensus, a classical Z-score sanity check, SHA-256 audit ledger). "An agent should be a PID controller, not a philosophy major writing a dissertation."
This is the thing only local can do. Because a local model is unlimited and has no per-call bill or rate limit, you can leave agents running around the clock. People run continuous security scans over their own codebase, watch a database for anomalies, and scrape the web every twenty minutes for opportunities, all 24/7. On cloud APIs that would cost more than the hardware; on local it is just electricity. A fleet of tireless workers that never sleeps, only possible when the model is yours.
Put a private mesh (Tailscale) across all your machines and your phone, and any device can use a model running on your home box from anywhere in the world, encrypted, never touching the public internet. Pair it with an orchestrator agent (Hermes or OpenClaw) that loads models and routes work across your machines for you, like an IT person you talk to in plain English. That is how a pile of computers becomes one personal AI cloud you own.
Produção em escala
k8s + AIBrix
The control plane. LLM-aware autoscaling on gpu_cache_usage_perc, distributed KV cache, KubeRay, high-density LoRA.
HA for a team
2x Ubuntu + MicroK8s, etcd on DRBD, Pacemaker VIP, LINSTOR, MetalLB, ArgoCD canary. Failover under 60s, serving Qwen3-Coder-30B at ~90 tok/s.
AI Coding Factory
Definition of Done as build gates: block merges below 85% coverage, security agent rejects vulnerable PRs. "Treat AI like an engineering org."