Detailed content
🎯 The right question: which model do you want to run?
Most people start in the wrong place: they look at the machine and ask, "what fits here?" Reverse the order. Start with the use case — the task determines the model size, the size determines memory needs, and memory points to the hardware. Hardware is the last question, not the first one.
🆕 New here?
"Use case" is the concrete task: summarize email, write code, classify text, chat. "Size" is the number of model parameters, measured in billions — the 7B, 13B, 70B of the name. More parameters ≈ more capable, but heavier. "Quantization" is how much each parameter has been compressed (Q4, Q8, FP16) — we’ll cover this in the next topic.
🔑 Three questions before any download
- •What task? Chat and drafting need little; production code and long reasoning need larger models.
- •What minimum quality will do? "Good enough" for most tasks comes well before the top of the line.
- •How often? Occasional use can tolerate slowness; a 24/7 agent needs to fit comfortably and run fast.
✓ For the use case (right)
- ✓"I want to summarize notes" → a 7B model is enough → fits in 8 GB
- ✓You know the memory ceiling before to download
- ✓Spend only what's necessary — no excess hardware
✗ Based on the hardware (wrong)
- ✗"I have a GPU, what's the biggest model?" → download a 70B that barely runs
- ✗Frustration: slow, freezing, doesn't fit properly
- ✗An expensive purchase for a problem a 7B model could solve
📐 The memory rule of thumb
This is the calculation that determines almost everything. The model's size on disk and in memory comes from a simple formula—you don't need more than multiplication:
🆕 New here?
"Weight" (or parameter) is a number the model learned. A 7B has 7 billion of them. "Bytes per weight" is how much space each number takes: FP16 stores it at full precision (2 bytes), quantize compresses to 8 or 4 bits (1 or ~0.5 byte), shrinking the file. "Context overhead" (the KV cache) is extra memory that grows as the conversation gets longer—reserve 1 to 3 GB for typical contexts.
Applying the formula to the most common sizes (weights only, before context):
| Model | Q4 (~0.6 GB/B) | Q8 (~1.0 GB/B) | FP16 (~2.0 GB/B) |
|---|---|---|---|
| 7B | ~4.5 GB | ~7.5 GB | ~14 GB |
| 13B | ~8 GB | ~14 GB | ~26 GB |
| 34B | ~20 GB | ~36 GB | ~68 GB |
| 70B | ~40 GB | ~70 GB | ~140 GB |
Read the table by choosing the line (the size your task requires) and the column (quantization). The number in the cell has to fit in your memory with room to spare — use up the context and leave some breathing room for the system.
💡 Practical tip
Never fill memory to the brim. Reserve 10–20% headroom for the operating system, 1–3 GB for the context, and prefer the largest quantization that still fits — Q4_K_M is the sweet spot: it shrinks a lot and loses little quality. A quick rule of thumb: size in billions ÷ 2 ≈ GB in Q4.
🧭 VRAM, RAM, or unified memory in your decision
The rule of thumb tells you how many GB. The remaining question is which memory. The type of memory your hardware profile offers determines which model range is worth using — and the speed you’ll experience.
🆕 New here?
VRAM is video card memory (GPU) — limited but fast; the model flies when it fits entirely in it. RAM is system memory, used by the CPU — plentiful and cheap, but slow for inference. Unified memory (Apple Silicon) is a single pool shared by the CPU and GPU — it fits a large model, with midrange bandwidth.
Read the diagram from left to right: each lane of free memory (blue) points to the widest model range (in cyan) that fits comfortably in Q4. With dedicated VRAM, this model flies; with the same amount of RAM or unified memory, it fits just as well—it just runs more slowly.
✓ Where each one excels
- ✓VRAM: the model fits entirely on the GPU → maximum speed
- ✓Unified: lots of cheap GB → runs huge models that no ordinary GPU can handle
- ✓RAM: the zero-cost path for small models on the CPU
✗ Where each one gets tight
- ✗VRAM: a little — exceed the limit, and it drops to the CPU
- ✗Unified: moderate bandwidth → 70B fits, but runs slowly
- ✗RAM: slow for inference; a large model turns into a turtle
⚡ Acceptable speed: tokens per second
Fitting is one thing; being usable is another. The metric that measures this is tokens/s — how fast the text comes out. And what matters here isn’t memory size, but band (how quickly memory delivers data to the processor).
🆕 New here?
"Token" is a piece of a word — on average, a word has 1 to 2 tokens. "Tokens/s" is how many of these chunks the model generates per second. For comparison: you read aloud at about ~5 tokens/s; above that, the text appears faster than you can read.
Use these ranges as a reference to see whether a model+hardware combination is worthwhile:
< 5 tokens/s — rough for chat
It works for batch tasks that run on their own (overnight, while you’re not watching), but chatting this way gets tiring. A typical sign that the model is too large for the CPU/RAM.
10–20 tokens/s — comfortable
The sweet spot for everyday chat: text comes out at reading speed. That's the realistic target for a 7B–8B in Q4 on a modest GPU or in unified memory.
40+ tokens/s — plenty of headroom
Near-instant responses, great for agents and high-throughput workflows. This happens when the model fits entirely in the VRAM of a high-bandwidth GPU.
💡 Practical tip
When the model fits entirely in fast memory, speed drops gradually as size increases. When overflows (part on the GPU, part on the CPU), performance drops sharply—it can fall from 30 to 3 tokens/s. That’s why the goal is: fit entirely in your fastest available memory, even if that means going one number smaller in the model.
💰 Start with what you already have
The best first machine is the one on your desk. The path to zero cost is straightforward: the laptop you already use + Ollama + a 3B to 8B model in Q4. Before thinking about buying anything, check what your machine has — it takes two commands.
⌨️ Practical example: read your machine's memory
Objective: find out how much VRAM and RAM you have — the two numbers that determine the largest model you can run. Paste this into the terminal.
How to check / estimate the largest viable model:
- Make a note of the free memory fastest one you have (VRAM if you have a GPU; otherwise RAM).
- Remove ~20% headroom for the system and 2 GB for the context.
- Divide what’s left by 0.6 → is the approximate maximum size in billions, at Q4. E.g.: 16 GB → ~11 GB usable → fits a 7B–8B with room to spare.
✓ What you already have is enough when…
- ✓The use case is chat, summarization, drafting, or classification
- ✓A 7B–8B in Q4 fits and runs at 10+ tokens/s
- ✓You’re still learning and testing the workflow
↑ Consider scaling up when…
- →The model required for the task doesn’t fit even in Q4
- →Speed is holding back your actual work
- →You’re going to leave an agent running 24/7
🚀 Putting together an upgrade plan
If it’s time to invest, spend in the right order. The golden rule: memory first, always. A model that doesn't fit won't run—having bandwidth doesn't help much if there's not enough space. Follow this priority:
More memory
It's what unlocks larger models. In RAM, it's usually cheap; in VRAM (more GPU), it's the quality jump that delivers the most. Resolve “will it fit?” before anything else.
Dedicated GPU (when speed becomes a bottleneck)
Only after it fits. A GPU with enough VRAM for the entire model multiplies tokens/s. Make up for bandwidth e capacity—small VRAM forces overflow to the CPU.
Multi-machine (later)
A large model on one machine, a fast model on another, all connected over the private network. That's the topic of module 2.4 (Tailscale) — don't buy with this in mind now.
🛒 Buying checklist
- •I’ve already defined the use case e o model size target?
- •The target fits in memory that I’m going to buy in Q4, with room for the OS and context?
- •A band does it deliver the tokens/s my usage requires?
- •O operating system supports my hardware without hassle (topic of 2.3)?
💡 Practical tip
Don’t buy for your dream model — buy for the model you actually uses today, with one tier to spare. The ecosystem gets more efficient every month: the hardware you buy will perform better tomorrow than it does now.
Self-check (optional): You have 16 GB of RAM and no dedicated GPU. Based on the rule of thumb, what’s the safest choice for everyday chat?
📌 Module summary
Next module:
2.3 — Operating system, drivers, and foundation