⚙️ Installation & Techniques
The hands-on heart of the course: install Ollama and run your first model, get a ChatGPT-like interface, learn to choose GGUF and quantization, adjust context and temperature, serve an OpenAI-compatible local API, and build a RAG from your documents.
Trail map
Detailed content
🚀 First model running: Ollama
From zero to your first chat with minimal friction — install, download, and run a local model.
Runtime that downloads, manages, and runs local models with one command; wraps llama.cpp.
It's the shortest path to your first result.
Runtime, llama.cpp, cross-platform.
On macOS/Linux, a single script; on Windows, a graphical installer.
No build or manual dependency setup.
curl install, ollama --version.
The command ollama run downloads the model and opens the prompt so you can chat.
Seeing the model respond proves the lab works.
ollama run, llama3.2, qwen2.5.
Commands to download, list, remove, and see what’s running.
Keeps disk and memory under control.
pull, list, rm, ps.
The :2b/:7b/:14b tags connect the model to the memory rule in Track 2.
Choosing wrong causes freezes or forces disk swapping.
Size tags, VRAM/RAM, start small.
A Modelfile is the recipe that defines the system prompt, parameters, and base model.
Create an assistant with a personality without programming.
Modelfile, SYSTEM, ollama create.
💬 ChatGPT-style interface: Open WebUI & LM Studio
From terminal to browser: chat with history, multiple models, and attachments.
Chat with history, model switching, and attachments, outside the terminal.
Brings the local experience closer to ChatGPT.
History, multiple models, attachments.
A front end that connects to Ollama or any OpenAI-compatible API, running locally.
It's the local community's default interface.
Front end, OpenAI-compatible, local.
A command docker run starts the interface on port 3000.
Installs in isolation, without cluttering the system.
docker run, port 3000, volume.
App that downloads GGUF and includes chat + a built-in local server.
A 100% graphical option for anyone who doesn't want to use the terminal.
GGUF, built-in server, desktop.
Configure the UI to use the Ollama endpoint and switch models.
It's the link between the polished interface and the engine.
Endpoint, model selection, system prompt in the UI.
Multiple chats, document RAG, prompt profiles, and users.
Turns the toy into a daily tool.
Multiple chats, RAG, profiles.
📦 Choosing and downloading the right model: GGUF and quantization
Where to find models, which format to download, and which quantization offers the best cost × quality.
Central repository of open models, with one model card per model.
That's where almost everything you'll run comes from.
Repository, model card, open models.
GGUF is the packaged format for running via llama.cpp/Ollama on CPU/GPU.
Downloading the wrong format won’t run in your runtime.
GGUF, safetensors, GPTQ/AWQ.
Suffixes like Q4_K_M, Q5, and Q8 indicate how much the weights have been compressed.
Q4_K_M/Q5_K_M are usually the quality × size balance.
Q4_K_M, Q5, Q8, sweet spot.
pull using the Ollama tag, or download the GGUF from HF and import it.
Each path serves a runtime and a level of control.
ollama pull, GGUF from HF, ollama create.
Recaps the memory rule for choosing the quantization based on VRAM/RAM.
Prevents choosing a model that won't fit.
Memory rule, decision flow, quantization based on VRAM.
Check the checksum and test with a standard prompt, comparing quants.
Too aggressive a quant degrades the response.
Checksum, standard prompt, compare quants.
🎛️ Parameters and quality: context, temperature, offload
The buttons that change the response and speed — and how to diagnose when something gets stuck.
num_ctx defines how much the model reads at once; long context uses memory (KV cache).
Increasing it without a plan can exhaust your memory.
num_ctx, KV cache, memory cost.
Temperature measures how random/creative the text is; low = deterministic.
Each task calls for a recipe (code = low, brainstorm = high).
temperature, top_p, top_k.
Defines the role, format, and constraints for the response.
It's the cheapest quality lever.
Role, format, constraints.
num_gpu determines how many layers fit on the GPU; the rest go to the CPU, which is slower.
Balancing offload is what makes it fit and run faster.
num_gpu, layers, GPU vs. CPU.
What speeds up or slows down generation, including KV cache quantization.
Small adjustments can double tokens/s.
KV cache, batch, tokens/s.
Cause checklist: model too large, aggressive quantization, huge context, swapping.
Solve it yourself without guessing in the dark.
OOM, quantization, context, swapping.
🔌 Serving through an OpenAI-compatible local API
Turn your local model into an endpoint that your apps, scripts, and agents can consume like OpenAI.
Expose the local model as if it were OpenAI for apps and scripts.
It's what connects the model to the agents in Track 4.
Endpoint, integration, reuse.
Ollama serves on localhost:11434, with native routes and OpenAI-compatible /v1 routes.
Knowing the right route prevents half the errors.
localhost:11434, /v1, routes.
One curl to stop /v1/chat/completions returns the response in JSON.
Confirm that the API responds before writing code.
curl, chat/completions, JSON.
The same OpenAI SDK, pointed at the local base_url.
Reuses the entire existing ecosystem.
OpenAI SDK, base_url, dummy api_key.
stream:true for real-time responses, an embeddings route, and model routing.
Cover real use cases beyond simple chat.
stream, embeddings, routing.
OLLAMA_HOST=0.0.0.0 + Tailscale to access it from another machine, without opening it to the public internet.
Remote access without leaving a security hole.
OLLAMA_HOST, tailnet, no public port.
🧠 Embeddings, memory & basic RAG
Give the model memory: retrieve relevant passages from your documents and answer based on them.
Vectors that represent the meaning of text; similar ones are close together in space.
It's the foundation of every similarity search.
Vector, similarity, semantic space.
RAG (Retrieval-Augmented Generation) searches for relevant passages and gives them to the model as context.
Answers questions about YOUR documents without retraining.
RAG, retrieval, context.
Ingestion (chunk → embed → store) and query (embed the question → find neighbors → assemble the prompt).
Understanding the flow is what lets you debug it.
Chunk, embed, neighbors.
Download a small embedding model and call the embeddings endpoint via API.
Embedding models even run on budget hardware.
nomic-embed-text, embeddings endpoint, lightweight.
A vector store (Chroma/FAISS/sqlite-vec) stores vectors for similarity search.
It's where RAG memory lives.
Chroma, FAISS, sqlite-vec.
Ingest a .txt, create embeddings, search, and respond with the local model.
Close out the track by proving the complete cycle.
Ingestion, search, verifiable answer.