Detailed content
⚖️ The master trade-off: memory × bandwidth
All local AI hardware faces the same tension. On one side, how much memory the machine has—that determines the size of the model that fits. On the other hand, the bandwidth of this memory — this determines the speed what the text sounds like. The four categories in this module are just different points on the same seesaw.
🆕 New here?
- Unified memory: RAM and VRAM become one shared pool for the CPU and GPU (Apple’s approach). Large models fit, but bandwidth is usually modest.
- VRAM: the memory dedicated that lives inside the graphics card. Usually smaller, but with extremely high bandwidth.
- Bandwidth (memory bandwidth): how many GB/s of memory bandwidth the chip gets. It's the #1 bottleneck for generation speed—more bandwidth means more tokens per second.
🔑 The central idea
Think in two independent dimensions:
- •Capacity (memory): the model fits or it won't fit. Without enough memory, it won't even load.
- •Bandwidth: once loaded, the bandwidth determines whether the answer comes out in a burst or drips out.
- •No one gets both at a low cost: Apple bets on capacity; GPUs bet on bandwidth. Choosing hardware means choosing where to compromise.
Read by comparing the two columns: the left, a much larger memory box (up to 512 GB) fed by a thin pipe—lots of capacity, low speed. At right, a smaller box (≈ 32 GB) with a wide pipe—low capacity, high speed. The thickness of the arrow é the bandwidth.
📊 The 4 categories at a glance
| Category | Memory | Bandwidth | Good for |
|---|---|---|---|
| Mac (unified) | high (up to 512 GB) | low | huge, slow models |
| AI Workstation | medium (~128 GB) | medium | balance, local dev |
| Powerful GPU | low-medium (32–96 GB) | extremely high | speed, until it fits in VRAM |
| Budget | low (8–24 GB) | low-medium | small models, embeddings |
🍎 Mac category: high unified memory, low bandwidth
O Apple Silicon (M-series chips in Mac Studio, Mac Pro, and MacBook) changed the game for local AI with a trick: unified memory. The GPU sees all the system RAM as if it were VRAM. A Mac Studio with 512 GB of unified memory can load a 200 GB+ model that no consumer GPU can handle. The trade-off is bandwidth: it's good for an integrated chip, but falls far short of a dedicated GPU — so the giant model runs, but slowly.
✓ Where the Mac excels
- ✓Runs models huge that don't fit on any consumer GPU
- ✓Quiet, cool, and energy-efficient for its size
- ✓macOS ready to go: MLX/Metal without installing any drivers
- ✓All-in-one box, no PC assembly or 1000 W power supply
✗ Where it gives way
- ✗Lower bandwidth → lower tokens/s that a dedicated GPU
- ✗Configurations that use a lot of unified memory are expensive
- ✗No CUDA: some projects assume NVIDIA and require adaptation
- ✗Soldered memory: you can’t upgrade it later
💡 Practical tip
If your goal is run the largest possible model in a single quiet device — and you’re fine with responses coming at reading speed, not jet speed — a Mac with lots of unified memory is hard to beat. If the goal is maximum speed for a model that fits in 24–32 GB, the next step up (GPU) is a better fit.
🏗️ AI workstations: medium memory, medium bandwidth
A new category emerged in the middle of the map: AI workstations — machines designed from the factory to run models, with plenty of unified/shared memory and better bandwidth than a regular Mac, without the price (and heat) of a GPU rack. It’s the balance for anyone who wants a large model e decent speed, usually running Linux.
NVIDIA DGX Spark (~128 GB)
A "desktop AI supercomputer" from NVIDIA with around 128 GB of coherent memory. It brings the CUDA stack in a compact form — good for prototyping medium/large models without going to the cloud.
AMD Ryzen AI Max "Strix Halo"
An AMD APU that allocates a large portion of system RAM as GPU memory — AMD’s answer to the idea of unified memory, in mini PCs and powerful laptops. Similar balance, ROCm ecosystem.
Typical OS: Linux
These machines almost always run Linux, where drivers (CUDA/ROCm) and AI runtimes are more mature. This is the category that calls for the most familiarity with the terminal—the subject of module 2.3.
🧭 How to position
The workstation sits exactly between the Mac (lots of memory, low bandwidth) and the GPU (little memory, high bandwidth): memory medium and bandwidth medium. It’s the choice for those who want neither extreme—they want a reasonably large model running at a reasonably good speed, without setting up a server.
🚀 Powerful GPUs: less VRAM, extremely high bandwidth
This is where the raw speed. A dedicated NVIDIA GPU has less memory than a top-of-the-line Mac, but its VRAM bandwidth is several times higher — so, as long as the model fits in VRAM, it generates tokens at a pace no other category can match. The golden rule changes: here, the limit isn't "which model fits on the machine," it's "which model fits in the board".
🖥️ Two GPU references
| GPU | VRAM | Bandwidth | Profile |
|---|---|---|---|
| RTX 5090 | ~32 GB | extremely high | top consumer, very fast |
| RTX 6000 (Pro) | ~96 GB | extremely high | professional, fits more, expensive |
✓ Pros
- ✓Tokens/s at the top—feels “real-time”
- ✓CUDA is the best-supported path across the ecosystem
- ✓Great for serving a fast API and agents that respond right away
✗ Downside
- ✗A model that doesn't fit in VRAM "spills over" to the CPU and slows to a crawl
- ✗High power consumption and heat; a robust power supply and case are required
- ✗Large VRAM (96 GB) is expensive — professional territory
💡 Practical tip
With a GPU, the game is keep everything within VRAM. A quantized model that takes up 28 GB flies on a 5090 with 32 GB; the same model at higher precision overflows, spills into RAM, and its speed drops like a rock. In module 2.2, you’ll learn to estimate whether it fits before downloading anything.
💵 Budget: the "everything else" still matters a lot
The fourth category is the biggest of all: Mac mini, laptops, and typical PCs — anything below the premium tier. Less memory (8–24 GB), modest bandwidth. But don’t confuse “budget” with “useless”: this is exactly where most people start, and running small models (Gemma, Phi, Qwen in 2–8B versions), embeddings, and classification, you can already handle a huge number of real-world tasks.
✓ What runs well
- ✓Small models (2–8B), quantized for chat and drafts
- ✓Embeddings and semantic search (RAG foundation) — lightweight
- ✓Classification, extraction, and summarization of short texts
- ✓Zero upfront cost: use the machine you already have
✗ Where it gets tight
- ✗Large models (30B+) won't fit or crawl
- ✗Very long context quickly runs out of memory
- ✗Without a GPU, the CPU can handle it—but at low tokens/s
💡 Practical tip
The best category for learn is the one you already have on your desk. Start on a budget with a small model and Ollama (Track 3), find out what you actually do with local AI, and only then decide whether it's worth moving up a tier. Expensive hardware bought before you know the use case often ends up gathering dust.
💻 Operating system in practice
The hardware category almost always brings an operating system along with it. There is no "best OS" in the abstract—there's the one that fits your machine and your comfort level. The three paths:
🧭 macOS × Linux × Windows
| OS | Works with | Pros | Cons |
|---|---|---|---|
| macOS | Apple Silicon | usability, MLX, no drivers | without CUDA |
| Linux | workstations / GPU | mature drivers, more control | terminal curve |
| Windows | Gaming GPUs | easy, WSL2 brings Linux along | more layers |
⌨️ Practical example: find your category
Objective: see how much memory (and what type) you have — the number that determines which of the 4 categories your machine falls into. Paste this into the terminal.
How to check: if the large number comes from free -h on a Mac, it’s unified (category 2). If there’s a VRAM separately in the nvidia-smi—that’s GPU (category 4). Not much memory and no dedicated GPU? Budget (category 5)—and it’s fine to start there.
Self-check (optional): which phrase correctly sums up this module's main trade-off?
📌 Module summary
Next module:
2.2 — How to choose what fits your machine