Detailed content
⚖️ Weights and parameters
When you see a model called "Llama 3.1 8B" or "Qwen2.5 32B", that number is the most important thing on the label. It tells you the brain size of the model. Understanding what it means already helps you choose better than most people.
🆕 New here?
Parameters (weights) are the numbers the model learned during training — they hold everything it "knows". "7B" means 7 billion parameters (the B stands for billion). "70B" = 70 billion; "235B" = 235 billion. In practice, the model file is basically a giant list of these numbers.
🔑 The rule to remember
More parameters usually means two things at once:
- •More capable: tends to reason better, make fewer mistakes, and handle difficult tasks.
- •Heavier: uses more memory and runs more slowly on the same machine.
- •It isn’t linear: a 70B model isn't “10× better” than a 7B model — the gains diminish, but memory costs rise along with them.
The most common ranges you'll find in the local AI world:
Small · 1B–4B
Gemma 2 2B, Llama 3.2 3B, Phi. Run on almost any laptop. Great for classification, short summaries, autocomplete, and simple tasks.
Medium · 7B–14B
Llama 3.1 8B, Qwen2.5 7B/14B, Mistral 7B. The “sweet spot” for everyday use: good at chat, writing, and simple coding, while still fitting on modest hardware.
Large · 27B–70B
Gemma 2 27B, Qwen2.5 32B, Llama 3.1 70B. More robust reasoning; they need a powerful GPU or a Mac with plenty of unified memory.
Huge · 100B+
DeepSeek-V3, Llama 3.1 405B, MoE models (mixture of experts). Top of the local lineup — they require workstation hardware or a high-memory Mac.
📊 Useful detail: “base” vs. “instruct”
Almost every model comes in two versions. The base only completes text. The instruct (or chat) was tuned to follow instructions and chat—it’s what you want 99% of the time. Look for the suffix -instruct or -chat in the name.
🔤 Tokens and context window
The model doesn’t read letters or entire words: it reads tokens. And it can only pay attention to a limited number of them at a time—the context window. These two concepts explain why the model sometimes "forgets" the beginning of a long conversation.
🆕 New here?
Token = a piece of text. It can be a short word, part of a word, or a punctuation mark. A rule of thumb for Portuguese/English: 1 token ≈ 4 characters ≈ ¾ of a word. The phrase "Local AI is power" has ~5–6 tokens. Context = how many tokens the model can “read at once” (input + response together).
The context window is measured in tokens. The most common sizes:
4,096 tokens — the basics
Fits a few pages. Enough for short questions and answers. Older models stopped here.
8,192 tokens — the comfortable option
Typical of many current models. Plenty of room to chat and paste a medium-length document without running into limits.
128,000 tokens — the long one
Llama 3.1, Qwen2.5, and others fit here. It can hold an entire book. But keep in mind: large context uses memory — you rarely need to use everything.
💡 The hidden cost of context
The larger the context window you enable, the more extra memory the model needs while it works (this is the KV cache, which you’ll see in topic 4). That’s why having “128k of context” isn’t free: loading everything can use up your memory. The best practice is to use only the context you need.
🗜️ Quantization: the trick that makes AI fit
This is perhaps the most important concept in the module. Quantization is what lets you run a large model on modest hardware. It’s the difference between “it won’t fit on my machine” and “it runs just fine.”
🆕 New here?
Quantization = reduce the accuracy of each weight so it takes up less space. Instead of storing each number with many digits (FP16, 16 bits = 2 bytes per weight), you store it with fewer (Q8 ≈ 1 byte; Q4 ≈ half a byte). It’s like saving a photo at lower quality: the file gets much smaller, and in practice, you barely notice the difference.
Look at the same model from 7B in three precisions. Notice how the size drops while quality barely declines:
Read from top to bottom: the file shrinks by half at each step (14 → 7 → 4 GB), while quality barely changes (100% → ~97%). That's why almost everyone runs quantized models — the memory savings are worth much more than the slight loss in quality.
✓ What you gain from quantization
- ✓The model fits in much less memory
- ✓Runs faster (less data to move)
- ✓Modest hardware can now run larger models
- ✓Q4/Q5 maintain excellent quality for everyday use
✗ The price to pay
- ✗A small loss of precision at each level
- ✗Over-quantizing (Q2, Q3) noticeably degrades quality
- ✗The loss is more noticeable on highly precise tasks (complex code)
- ✗Below Q4, it’s rarely worth it
💡 Practical tip
When in doubt, start with Q4_K_M or Q5_K_M — these are the community-recognized “sweet spot”: they fit in less memory while keeping quality very high. You’ll see the full suffixes (Q4_K_M, Q5_K_S…) in detail in Track 3, module 3.3.
🧠 Memory: the feature that matters most
If there’s one number that determines whether a model will run on your machine, it’s memory. It’s not the processor type or the graphics card brand: it’s how much memory you have to load the weights. Here you’ll learn to estimate that in your head.
🆕 New here?
VRAM is video card memory (GPU) — the fastest kind for AI. RAM is system memory. On Macs with unified memory, they're the same thing. The model needs to fit entirely (or almost entirely) in this memory to run fast. The KV cache is extra working memory that the model uses to “remember” what it has already read in the conversation—and it grows along with the context.
The rule of thumb is straightforward:
memory (GB) ≈ parameters (B) × bytes per weight + context headroom
Where bytes per weight = 2 (FP16), 1 (Q8) or ~0,5 (Q4). Leave ~10–20% headroom for the system and KV cache.
Applying the formula, here’s the table worth sticking on the wall—memory estimate only the weights (add some headroom):
| Model | Q4 (~0.5 B/weight) | Q8 (~1 B/weight) | FP16 (2 B/weight) |
|---|---|---|---|
| 7B | ~4 GB | ~7 GB | ~14 GB |
| 13B | ~7 GB | ~13 GB | ~26 GB |
| 34B | ~19 GB | ~34 GB | ~68 GB |
| 70B | ~39 GB | ~70 GB | ~140 GB |
Practical reading: with a GPU of 8 GB you’re comfortable running a 7B in Q4. With 24 GB, a 34B in Q4 comes in. A 70B only fits entirely on large hardware (48 GB+ GPU or Mac with plenty of unified memory). Always leave some headroom — the context uses memory.
🧩 The KV cache, in one sentence
In addition to the weights, reserve memory for the KV cache: it keeps the conversation’s “work in progress” and grows with the number of tokens in the context. Doubling the context can double that cost. So, on limited hardware, reducing the context window is sometimes what finally makes the model fit.
💡 Practical tip
If the model doesn’t fit entirely in the GPU, you can send part of it to RAM (offload)—it runs, but more slowly. Better than not running at all. Track 2 (Hardware) and module 3.4 show you how to adjust this in practice.
🌳 Open model families
The open world isn’t just one model — it’s families, each from a different lab, with different strengths. Knowing the main names saves you from getting lost when it's time to choose. These are the ones you'll come across most often:
🦙 Llama (Meta)
The most popular family, with the largest ecosystem. Sizes from 1B to 405B. A strong general-purpose model, with plenty of materials and tools around it. Its own license (the "Llama Community License")—open, with some restrictions.
🐉 Qwen (Alibaba)
Highly competitive Chinese lineup, from 0.5B to 72B+. Excellent at coding and multilingual tasks (including Portuguese). Many variants under the Apache 2.0 license—very permissive.
💎 Gemma (Google)
Highly efficient small and medium models (2B, 9B, 27B) — they punch above their weight. Great for modest hardware. Their own license allows broad use.
🌬️ Mistral (Mistral AI)
European lab. Mistral 7B and Mixtral (MoE) became famous for delivering a lot with little. Many are under Apache 2.0; some premium models have a restrictive license.
🐳 DeepSeek
Chinese lab that took the world by storm with large reasoning and coding models (DeepSeek-V3, R1). At the top of open model capabilities — but the largest ones require serious hardware.
🔬 Phi (Microsoft)
Small models (3B–14B) trained on extremely high-quality data. They punch above their weight. MIT license in several versions — among the most permissive.
🎯 How to navigate among so many
- •Small tier (≤4B): Gemma 2 2B, Phi, Llama 3.2 3B, small Qwen models — for classification, summaries, and low-powered machines.
- •Everyday tier (7B–14B): Llama 3.1 8B, Qwen2.5 7B/14B, Mistral 7B — everyday chat and coding.
- •Heavy tier (27B+): Gemma 2 27B, Qwen2.5 32B, Llama 70B, DeepSeek — serious reasoning, requires powerful hardware.
- •Embeddings (topic in Track 3): tiny specialized models (e.g., nomic-embed) — a different category, for memory/RAG.
💡 Practical tip
There is no “best model” — there is the best one for your task on your hardware. A good routine: pick 2–3 candidates that fit your memory and test them all with the same real prompt of yours. In practice, the difference usually matters more than any benchmark ranking.
📜 Licenses matter
"Open" doesn’t mean "anything goes." Each family comes with a license that determines what you can do—especially if you're using it for work or a product. Ignoring this can cause headaches later.
🆕 New here?
License = the model’s “terms of use.” It says whether you can use it commercially, redistribute it, modify it, and what conditions you need to meet. The most permissive are Apache 2.0 e MIT (almost “do whatever you want”). Others, like the Llama license, are open but have constraints (e.g., rules for very large companies or attribution requirements).
✓ Permissive (Apache 2.0 / MIT)
- ✓Commercial use allowed, no hidden catches
- ✓Can modify and redistribute
- ✓E.g.: many Qwen models, Mistral 7B, Phi, embedding models
- ✓The smoothest choice for a product
✗ Restricted / custom
- ✗May limit commercial use at a certain scale
- ✗May require attribution or acceptance of terms
- ✗Some prohibit specific uses
- ✗E.g., Llama license, some "research only" models
Where does all this live? In model card from Hugging Face — each model's page. Knowing how to read it is the practical skill in this module:
⌨️ Practical example: read a model card on Hugging Face
Objective: open a model’s page and extract, in 30 seconds, everything you need to decide whether it’s right for you. Open it in your browser (no download needed).
How to check: if you were able to say "it’s a 7B, 128k context, Apache 2.0, has GGUF Q4_K_M, instruct version"—you’re reading a model card. These five items are exactly what you need to decide whether it fits and whether you can use it.
✅ Checklist before adopting a model
- →Does it fit? Parameters × quantization must fit in your memory (topic 4).
- →Can they? The license permits your use (personal vs. commercial).
- →Does it work? It's the instruct version and has the context you need.
- →Is there a ready-made option? A quantized GGUF is available to download (Track 3).
Self-check (optional): you want to run a 13B and has a GPU with 8 GB. What’s the most sensible move, according to this module?
📌 Module summary
Next:
You’ve completed Track 1—Fundamentals. The theory is in place; now it’s time for hardware. Track 2 — Hardware & Lab: machine categories, what fits yours, and how to set up your lab.