Detailed content
📏 Context window in practice
A context window is how much the model “reads at once” before responding — your question, the chat history, and any text you paste in. If the window is small, the model “forgets” the start of the conversation. If it’s too large, it uses memory unnecessarily.
🆕 New here?
"Context window" = the token limit for what goes in at once (e.g., 8192 = ~6,000 words). In Ollama, this is the num_ctx. "Token" = part of a word; ~4 characters on average. "KV cache" = the working memory the model builds for each context token — it grows along with num_ctx and is what makes long contexts weigh heavily on VRAM.
Choosing the window size is a three-step balancing act:
Start with the default
Many models come with num_ctx of 2048 or 4096. For chat and short questions, that’s enough — don’t change it just for the sake of changing it.
Increase it when the task calls for it
Going to paste in a long document, large codebase, or extensive history? Increase it to 8192, 16384, or more — as long as the model supports that limit.
Pay the price in memory
Doubling the context window ≈ doubling the KV cache. If VRAM runs out, the model offloads layers to the CPU and slows down — or simply out of memory.
💡 Practical tip
Don’t turn on 128k context “just in case.” If you usually ask short questions, a 4k–8k window responds faster and frees up VRAM for layers of the model — which matters most for quality.
🌡️ Temperature, top_p, and top_k
For each token, the model has several candidate words. Temperature, top_p e top_k determine how much risk it takes when choosing. These three controls manage the precise ⟷ creative.
🆕 New here?
- Temperature — how “creative/random” the model is. Low (0–0.3) = deterministic and repeatable; high (0.8–1.2) = diverse and surprising.
- top_p (nucleus) — cuts off the tail: considers only the words that add up to, for example, 90% of the probability (top_p 0.9).
- top_k — limits the choice to k most likely words (top_k 40). Adjust temperature first; top_p/top_k are fine-tuning.
📋 Recipes by task
| Task | temperature | top_p | Why |
|---|---|---|---|
| Code / SQL | 0.1–0.2 | 0.9 | accuracy and reproducible answers |
| Extraction / JSON | 0.0–0.2 | 0.8 | rigid format, with no invented fields |
| Chat / explanation | 0.5–0.7 | 0.9 | natural, but still focused |
| Brainstorming / creative | 0.9–1.2 | 0.95 | diversity, unexpected ideas |
✓ Low temperature works for
- ✓Generate code that needs to compile
- ✓Extract data in a fixed format
- ✓Answers that should always be the same
- ✓Classification and routing
✓ High temperature works for
- ✓Generate many variations of an idea
- ✓Marketing copy and headlines
- ✓Break repetition and stiff-sounding responses
- ✓Fiction and scripts
🎭 System prompt: directing the model
Before asking any questions, the system prompt tells the model who it is e how it should respond. It’s the cheapest quality lever there is: when written well, it fixes verbosity, format, and tone without making you switch models.
🆕 New here?
"System prompt" is a fixed instruction, separate from your message, that applies to the entire conversation. Think of it as the employee’s “job and rules”: it defines paper (what it is), format (as delivery) and constraints (what it cannot do).
🧱 The three building blocks of a good system prompt
- •Role: "You are a concise technical assistant for developers."
- •Format: "Answer in no more than 5 lines. Use bullets when listing steps."
- •Restrictions: "If you don't know, say so. Don't make up commands."
⌨️ Practical example: a "concise technical assistant"
Objective: a reusable system prompt that you can paste into any model (Ollama, Open WebUI, API) for short, direct answers. Save it as system.txt or use it in the Modelfile.
How to use it in Ollama (Modelfile):
How to check: ask the same question in the base model and in the technical. The second one should respond much more briefly and directly, without the “Sure! Here’s...”
🧩 GPU/CPU offloading and layers
A model is a stack of layers. Ideally, they all fit in the GPU, where they run fast. When they don’t fit, the runtime overflows the ones left for the CPU — and those run much slower. The parameter that controls this is the num_gpu.
🆕 New here?
"Offload" = push part of the GPU’s work to the CPU. "num_gpu" (in Ollama) = how many model layers are placed on the GPU; the rest goes to the CPU/RAM. More layers on the GPU = faster, since fit in VRAM. If you force too many layers and VRAM runs out, it becomes out of memory.
Read from left to right: the 7 first layers (purple) fit on the GPU and fly; the 3 left over (cyan) spill over to the CPU and drag down speed. Increasing num_gpu moves the boundary to the right — until the VRAM fills up.
✓ Balanced offloading
- ✓All (or almost all) layers fit on the GPU
- ✓There’s VRAM left for the context’s KV cache
- ✓Chose a quant (Q4_K_M) that fits comfortably
- ✓stable and predictable tokens/s
✗ Poorly tuned offloading
- ✗Half the layers ended up on the CPU
- ✗num_gpu set too high → out of memory
- ✗Model too large for the available VRAM
- ✗Speed drops with every token
💡 Practical tip
Ollama tries to detect offloading automatically — in most cases, leave it on automatic. Only adjust num_gpu manually if you want to fit a slightly larger model (by deliberately offloading a few layers) or if you’re running two things on the same GPU.
⚡ KV cache, batch processing, and performance
Speed isn't just "a good GPU." Three things affect tokens/s: how much memory the KV cache uses, the batch size, and where the layers are. Here you bring all the parameters together in one place—via API.
📊 What speeds things up × what slows them down
- ✓Model fitting entirely in the GPU
- ✓Context window only as large as needed
- ✓KV cache quantization (q8/q4) to free up VRAM
- ✓Keep the model “hot” (loaded) between calls
- ✗Layers spilling over to the CPU
- ✗giant num_ctx “just in case”
- ✗Swapping to disk when RAM fills up
- ✗Reload the model with every request
🆕 New here?
"Batch" = how many tokens the model processes at once during the prompt-reading phase; a larger batch speeds up long prompts but uses more memory. "KV cache quantization" = store working memory at lower precision (e.g., q8_0) to fit a larger context window in the same VRAM—with minimal loss.
⌨️ Practical example: parameters via the Ollama API
Objective: run a controlled generation, passing temperature and context window together in the field options. Paste this into the terminal (Ollama needs to be running).
How to check: run the same command twice. With temperature:0.2 the responses come out nearly identical (deterministic). Switch to 1.0 and repeat: now they vary much more. The field eval_count / eval_duration in the JSON response gives you the tokens/s.
🩺 Diagnosis: why is it slow or bad?
When the output disappoints, the problem is almost always one of four suspects: a model that's too large, overly aggressive quantization, an oversized context, or disk swap. This checklist separates “it's slow” from “it's bad.”
✗ Symptom: it's SLOW
- ✗Model too large → layers on the CPU
- ✗huge num_ctx → KV cache ran out of VRAM
- ✗Full RAM → disk swapping
- ✗Model reloading on every call
✓ How to fix SLOW
- ✓Drop down one size (14B → 7B) or one quantization level
- ✓Reduce num_ctx to what the task requires
- ✓Close apps that use up RAM; check with ollama ps
- ✓Keep the model loaded (keep-alive)
✗ Symptom: it's BAD
- ✗Aggressive quant (Q2/Q3) → fragile responses
- ✗High temperature for a task that calls for precision
- ✗No system prompt → the model goes off the rails
- ✗Short context → it forgets the beginning
✓ How to fix BAD
- ✓Move up to Q4_K_M/Q5_K_M (the sweet spot)
- ✓Lower the temperature to 0.1–0.2
- ✓Add a clear system prompt (topic 3)
- ✓Increase num_ctx enough for the conversation
Self-check (optional): you need to generate SQL that's always the same and correct. Which setting makes the most sense?
📌 Module summary
Next module:
3.5 — Serving via a local API