Detailed content
🤗 Hugging Face: the “GitHub of models”
Before running anything, you need a place to get the models. That place is the Hugging Face: the central repository where almost every open model lives. Knowing how to read a model page there answers 80% of your “which one should I download?” questions.
🆕 New here?
Hugging Face (the “HF”) is like a GitHub just for AI: a site where labs and the community publish open models, each on its own “model page” (the model card). O model card is the model’s README — it describes what the model is, how many parameters it has, what license it uses, and lists the files to download.
🔑 What to look for in the model card
Open the model page and look for four things — they determine everything:
- •Parameters: is in the name (7B, 14B, 70B). More parameters ≈ more capable and heavier.
- •Variant: prefer the version Instruct / Chat (trained to chat), not the raw “base” model.
- •"Files" tab: the actual files. This is where you can see if there is
GGUFand which quantizations. - •License: Open source ≠ free to use for anything. Check commercial use before relying on it.
📊 Model-finding shortcut
- Look for repositories with "GGUF" in the name (e.g., published by bartowski or by the lab itself)—they're already ready for Ollama/LM Studio.
- Sort by downloads/trending: what the community uses tends to be stable and well tested.
- A good, popular model to start with today: Qwen2.5, Llama 3.1/3.2, Gemma 2, Mistral.
📦 Formats: GGUF, safetensors, GPTQ/AWQ
In the "Files" tab, you'll come across different extensions. They aren't "better" or "worse" versions—they're packaging for different runtimes. Choosing the wrong one is the #1 cause of "I downloaded it and it won't run.".
🆕 New here?
GGUF = a single-file format that packages the model ready to run on the CPU and/or GPU via llama.cpp — the engine behind Ollama and LM Studio. It’s the format for “plug-and-play” local AI.
safetensors = a “raw” weights format from the Python/PyTorch ecosystem (Transformers, vLLM). Safe and standard for training and GPU servers — but requires more setup.
GPTQ / AWQ = already quantized weights made to run only on GPU via runtimes such as vLLM/AutoGPTQ. Fast on GPU, but not the easiest way to get started.
✓ GGUF — the course's choice
- ✓Runs on Ollama and LM Studio without any setup
- ✓Runs on CPU, GPU, or both (offload)
- ✓A single file, with quantization built in
- ✓Ideal for Mac, laptop, and PCs without a monster GPU
✗ safetensors / GPTQ / AWQ
- ✗Require Python + Transformers/vLLM and dependencies
- ✗GPTQ/AWQ generally require a dedicated NVIDIA GPU
- ✗They don’t connect directly to Ollama/LM Studio
- ✗They shine on servers and in training—not in “easy to get started”
💡 Practical rule
For everything in this course (Ollama, LM Studio), the answer is almost always GGUF. Only switch to safetensors/GPTQ/AWQ when you’re serving at scale on a dedicated GPU — that’s covered in Track 4. When in doubt: if GGUF is available, download the GGUF.
🔢 Reading the quantization levels
There are lots of variants inside GGUF: Q4_K_M, Q5_K_M, Q8_0... This is the heart of the choice. Deciphering these suffixes is what separates "downloaded the right one" from "downloaded 14 GB that won’t fit".
🆕 New here?
Quantization = reduce the precision of the model’s weights (from 16 bits to 8, 5, 4...) so the file is much smaller and fits in less memory, with little loss in quality.
Reading Q4_K_M: o Q4 = ~4 bits per weight (the lower the number, the smaller and "more compressed"); the K = a modern “k-quant” method (smarter than the old one); the M = size Medium of the variant (there are S/small e L/large). FP16 = no quantization, full precision.
Use a model 7B as a benchmark. See what each quantization does to size and quality — the sweet spot is where the file is already small but quality has barely dropped:
| Quant | Bits/weight | Size (7B) | Quality | When to use |
|---|---|---|---|---|
| FP16 | 16 | ~14 GB | maximum | reference / spare VRAM |
| Q8_0 | ~8 | ~7.7 GB | ~almost identical | wants near-perfect quality |
| Q6_K | ~6.5 | ~5.5 GB | great | upper-middle ground |
| Q5_K_M | ~5.5 | ~5.1 GB | very good | high sweet spot |
| Q4_K_M ★ | ~4.5 | ~4.4 GB | good | best balance (default) |
| Q3_K_M | ~3.5 | ~3.5 GB | starts to drop | only if memory runs out |
| Q2_K | ~2.6 | ~2.8 GB | degrades quite a bit | last resort |
Read from top to bottom: the file shrinks quickly (from ~14 GB to ~4.4 GB) while quality drops slowly — even the Q4_K_M, where the curve is still high. Below it (Q3, Q2), quality starts to drop sharply. That’s why the community treats Q4_K_M / Q5_K_M as the default balance.
💡 Practical tip
Rule of thumb: start at Q4_K_M. If you have memory to spare, move up to Q5_K_M or Q8_0 and gain a little quality. Only drop to Q3/Q2 if the model doesn't fit — and knowing you’ll notice the loss.
⬇️ Downloading: via Ollama vs. directly
There are two paths: let the Ollama download it for you (the easiest option) or get a GGUF directly from Hugging Face and register it on your machine. You'll use both — start with the simple one.
⌨️ Option 1 — Ollama downloads it for you
Objective: download Qwen2.5 7B in the right format and quantization, with one command. The q4_K_M in the tag, choose the sweet-spot quantization.
How to check: o ollama list shows the line qwen2.5:7b-instruct-q4_K_M with ~4.4 GB. Run ollama run qwen2.5:7b-instruct-q4_K_M and ask a question—if it answers, it worked.
⌨️ Option 2 — GGUF directly from Hugging Face
Objective: use a GGUF that no is in the Ollama catalog (downloaded from the "Files" tab of a model card). Three steps:
Download the .gguf file
In the model card, the Files, download the desired quant (e.g., ...Q4_K_M.gguf).
Create a Modelfile
A text file that only points to the GGUF with the line FROM.
Log and run
ollama create turns the GGUF into your own model, with a name.
LM Studio (no terminal): in the search tab, find the model and choose the quantization (e.g., Q4_K_M) and click Download. It recognizes the GGUF and opens the chat—no commands.
How to check: ollama list shows my-qwen and the chat responds. In LM Studio, the model appears in “My Models” and loads in the chat.
💡 Ollama shortcut
Ollama also pulls GGUF files directly from Hugging Face, without a Modelfile: ollama run hf.co/usuario/repo-GGUF:Q4_K_M. Replace it with the repository path and the quantization you saw in the “Files” tab.
🧮 Matching model × hardware
Quant chosen, one final question remains: Will it fit on my machine? The rule of thumb from Track 2 completes the loop — the file size needs to fit in memory with room to spare for the context.
Approximate size per model × quantization (and how much memory you need to run it well):
| Model | Q4_K_M | Q8_0 | Ideal memory |
|---|---|---|---|
| 7B | ~4.4 GB | ~7.7 GB | 8–12 GB |
| 13B | ~7.9 GB | ~14 GB | 12–16 GB |
| 34B | ~19 GB | ~36 GB | 24–40 GB |
| 70B | ~40 GB | ~75 GB | 48–80 GB |
The difference between "fits" and "doesn't fit" is the difference between fast and unusable. Here are the two scenarios:
✓ Quant fits in memory
- ✓The entire model fits in VRAM/RAM
- ✓Responses are smooth, with many tokens/s
- ✓There’s room left for the context (KV cache)
- ✓You can run it 24/7 without freezing the machine
✗ Out of memory
- ✗Part goes to the CPU/disk (swap)—extremely slow
- ✗Low tokens/s, sluggish responses
- ✗OOM (out of memory) error or freeze
- ✗Answer: download a smaller quantization or a smaller model
💡 Decision flow
How much memory do you have? → choose the largest model whose Q4_K_M fit with 2–4 GB to spare. If it fits with room to spare, increase the quantization (Q5/Q8) before to moving up to a larger model — extra quality is cheaper than doubling the size. 8 GB ≈ 7B Q4; 16 GB ≈ 13B Q4 or 7B Q8; 24 GB+ ≈ 34B Q4.
✅ Checking quality and integrity
Downloaded — now confirm two things: that the file came entire (an uncorrupted download) and that the quantization didn't go too far. Two minutes of testing can save you hours of "why is this model so bad?".
⌨️ Practical example: checksum + standard prompt
Objective: verify the file’s integrity and run a sanity check on the response.
How to check: The SHA256 must match the one on the page. The response should be coherent, in Portuguese, with 3 genuine bullets—without repeating, getting stuck in a loop, or switching languages. If it turns into a mess, the quant may be too aggressive: move up to Q5_K_M or Q8.
✓ Healthy quant
- ✓SHA256 matches the HF page
- ✓Follows what you requested (format, language)
- ✓Coherent text, without strange repetitions
- ✓Similar answers in Q4 and Q5 (minimal loss)
✗ Too much quantization
- ✗Word salad or meaningless phrases
- ✗Ignores the request / switches languages on its own
- ✗Loop: repeats the same sentence endlessly
- ✗Gets simple calculations/facts wrong that Q5 gets right
Self-check (optional): You have 8 GB of VRAM and want to run a 7B model with the best balance of quality and size. Which quantization is the “sweet spot” to start with?
💡 Practical tip
Keep one of your own “standard prompts” (a summary, a calculation, a question about something you know) and run it whenever you test a model or quant. Compare the same a question between versions is the fastest and most honest way to feel the quantization loss.
📌 Module summary
Next module:
3.4 — Parameters and quality: context, temperature, offload