RAG has two halves that live in separate workflows. One prepares the foundation: you index the documents (break them into chunks, turn them into numbers, store them in a vector store). The other answers: the question goes through a classifier that decides whether it's worth using the agent; the agent searches the vector store for the relevant passage and responds grounded in this section — and, when it finds nothing, it falls back to a fallback be honest instead of making things up.
Illustrative diagram — indexing (blue) is separate from querying (cyan). The agent searches for the passage; when nothing is relevant, it follows the fallback branch instead of making things up.
❓ The problem: AI doesn’t know what isn’t in the context
What it is
An AI model only answers about two things: what it learned in training (general knowledge, with a cutoff date) and what's in the context from that conversation (what you pasted there). It has no magical access to the your files. If you don’t put the exchange policy, FAQ, or manual within its reach, it simply doesn’t know them.
What is it?Context — the text “window” AI can see in a conversation: instructions, messages, and any document you attached. RAG — retrieval-augmented generation: the pattern of retrieving the right passage from your data and injecting it into the context before the AI generates a response.
The danger is subtle: without the source at hand, AI doesn’t stay silent — it tends to fill in the gap, making up a deadline, an amount, or a rule that seems plausible. That's hallucination. RAG exists precisely to close this gap: instead of hoping the model will "remember" your policy, you make sure the policy text is in context at the right time. This is the reason of everything that follows in this module.
Why learn
Understanding this limit helps you stop expecting the impossible from AI and start giving it the right information. Instead of "why doesn't it know my policy?", the question becomes "how do I make my policy available to it?".
- •No source, no truth: the AI doesn't make things up on purpose — it fills in gaps.
- •Your data = within reach: policies, FAQs, and manuals need to be included in the context.
- •RAG fundamentals: retrieving the source is what grounds the response.
Key concepts
The AI only sees the context.
Your files don't go in automatically.
Without a source, it fills in the blanks.
Retrieves and grounds the answer.
⚖️ System prompt vs. vector store (when to use each)
What it is
There are two ways to give the agent knowledge. In the system prompt, you paste the entire text directly into the instructions—the simplest and cheapest approach, great for a small, stable document. In vector store (RAG), the agent searches only the relevant sections for each question—scalable and recommended when the knowledge base is large or growing.
What is it?System prompt — the fixed block of instructions that defines how the agent acts; it can contain text, but always takes up context. Vector store — a repository that stores your documents, ready for search by meaning; the agent retrieves only the necessary piece.
Why learn
Choosing the wrong option is costly in two ways. If you put a huge knowledge base in the system prompt, you use up context on every call (more tokens, slower) and still hit the context window limit as the knowledge base grows. If you set up a vector store for a single paragraph, you add complexity without any benefit. The rule of thumb: small, fixed document → system prompt; large or growing knowledge base → vector store.
✓ Vector store (choice for scale)
- ✓Searches only the relevant section — it doesn't load the entire database.
- ✓Scale: you can grow from 5 to 500 documents.
- ✓Lower context cost per question.
✗ Everything in the system prompt (doesn’t scale)
- ✗Loads the entire document on every call.
- ✗Consumes context (tokens) even for things you didn't ask about.
- ✗Limits growth: the window fills up as the knowledge base grows.
🎯 Practical tip
Start with the simplest option: if it’s a one-paragraph FAQ, paste it into the system prompt and test it. Move to a vector store only when the knowledge base has several sections or starts to grow — don’t build an indexer for something that fits in three lines.
Key concepts
Direct, inexpensive, short text.
Finds what's relevant, scales well.
A full prompt uses tokens every time.
Fixed and short → prompt; large → vector.
🧬 Embeddings & vector store in plain language
What it is
Embedding is turning text into numbers that capture the meaning. Phrases with similar meanings map to nearby numbers, even when they have no words in common. The vector store stores these numbers and, when you search, finds the passages whose meaning most closely matches the question—not by matching exact words, but by similarity of meaning.
What is it?Embedding — a list of numbers (a “vector”) that represents the meaning of a text. Similarity search — find the passages whose vectors are closest to the question vector. That's why "how do I return a product?" finds a document titled "Exchange Policy," even without the word "return".
Illustrative diagram — the search compares distances in meaning space. The points near the question (blue) become the retrieved passages; the far ones (gray) are ignored.
Why learn
This is the engine that makes search "understand" the question. Without embeddings, you'd depend on the user using exactly the same words as the document—fragile and prone to failure. With embeddings, "I want my money back" finds the "Refund Policy" because the meaning matches. Knowing this explains why the vector store finds the right passage and why it sometimes gets it wrong (when the meaning really is distant).
Key concepts
Embedding captures meaning.
Distance measures similarity.
Search by meaning, not by letter.
It’s what makes RAG “understand.”
📥 Indexer (manual trigger → docs → vector store)
What it is
O indexer is a flow separate of the agent, which exists only to populate the vector store. The sequence is: manual trigger → read the documents → split into pieces (text splitter) → generate embeddings → populate the vector store. You run this workflow once to create the foundation, then again whenever the documents change.
What is it?Manual trigger — you press “run” when you want; it doesn’t run on its own. Text splitter — splits a long document into smaller chunks, sized for search to compare effectively. In-memory vector store — a version that lives while the workflow is active; great for learning and testing.
Illustrative diagram — the indexer is a one-way flow that ends by populating the vector store. It doesn’t answer questions; it only prepares the knowledge base.
Why learn
Separate the indexing of the queries is the standard RAG architecture pattern. The indexer runs rarely (once, or when the knowledge base changes); the agent runs with every question. Mixing the two would make you re-index everything with every query—slow and expensive. Understanding this separation helps you avoid the most common mistake: activating the agent before the knowledge base exists.
🎯 Practical tip
Run the indexer before before activating the agent — the database needs to be populated for the search to find anything. If you use an in-memory vector store, remember: when you restart the workflow, the database disappears; run the indexer again before testing.
Key concepts
Indexing ≠ querying.
Text splitter prepares the search.
Once or when updating.
The output is the populated database.
🚏 Classifier — route before using the agent
What it is
O text classifier is a node that decides early, right at the start, whether the message is relevant (e.g., "is this a support question?" vs. "is it spam / out of scope?") and routes accordingly. Only what matters moves on to the more expensive agent; the rest is stopped or handled cheaply before spending on the model.
What is it?Classify — fit the input into a category (“support,” “sales,” “out of scope”). Route — route each category to a different path in the flow. Triage — this inexpensive filter at the entry point, before the heavy work.
Illustrative diagram — cheap triage decides where to go. Only the "relevant" path (cyan) reaches the expensive agent; everything else stops first, saving tokens.
Why learn
A low-cost triage before of the agent saves tokens and avoids out-of-scope responses. Without the classifier, every message — including spam and topics that aren't even yours — triggers the expensive agent, which uses up context and may even try to answer something it shouldn't. With it, you use the model only where it's worthwhile and keep the agent within the agreed scope.
Key concepts
Triage at the input.
Each category, a path.
Only relevant content uses up the agent’s capacity.
Blocks off-topic content.
🎯 Confidence score & fallback (don’t make things up)
What it is
The agent emits a confidence marker along with the response — for example, high (>0.8) / medium (0.6–0.8) / low (<0.6), or even a simple label like “policy found” / “no policy found.” A code node reads this marker and a IF node branches: when there’s a relevant passage, it follows the response draft; when there isn’t, it falls back to a fallback politely (“we’ll look into it and get back to you”) instead of hallucinating.
What is it?Confidence score — a note the agent gives to indicate how well the answer is supported by the source. Fallback — the honest alternative when there isn’t enough information in the knowledge base. IF node — the branch that chooses between “respond” and “fallback.”
Illustrative diagram — the confidence fork. The low-confidence branch (red) isn't an error: it's the agent refusing to make things up.
Why learn
An agent that refuses to make things up is a result desired, not a bug. When it falls back because it couldn't find the policy, it's being honest—which is far better than giving an invented deadline or amount that could cause losses and break trust. The confidence threshold is what makes this behavior controllable: you set the threshold and decide in code when to respond and when to back off.
✓ Refuse to make things up (honest fallback)
- ✓No relevant passage → "we don't have that here; let's research it and get back to you."
- ✓Low confidence triggers the fallback automatically.
- ✓Predictable and auditable behavior (you can see the score).
✗ Make up an answer
- ✗Make up a deadline or amount that "seems right."
- ✗Answer confidently even without a source in the store.
- ✗Treating the agent’s “I don’t know” as a defect to be removed.
The RAG build sequence
-
1
Index — run the indexer (docs → chunks → embeddings → vector store) and confirm the knowledge base is populated.
-
2
Classifier — put triage at the entry point to block anything out of scope before it reaches the agent.
-
3
Agent with vector store tool — the agent consults the knowledge base and responds based on the retrieved passage.
-
4
Fallback — read the confidence, branch in the IF, and, if there’s no relevant excerpt, respond with the honest fallback.
🎯 Goal: build the knowledge base (indexer) and the agent that queries it
crie um indexador (trigger manual → ler <meus-documentos> → dividir em pedaços → embeddings → vector store) e um agente que classifica a pergunta, busca no vector store e, se não achar nada relevante, responde com um fallback honesto em vez de inventar.
✅ How to verify: run the indexer once; ask an out-of-scope question and confirm that the agent falls back (doesn’t hallucinate).
🧩 Pseudocode: read the confidence marker and branch in the IF
confianca = ler_marcador(resposta_do_agente) // ex.: 0.42
se confianca < 0.6:
rota = "preciso pesquisar" // fallback honesto
senão:
rota = "rascunhar resposta" // responder ancorado
// o nó IF usa "rota" para escolher o caminho do fluxo
✅ How to verify: force low confidence (a question with no source) and confirm that "route" becomes "need to research."
🎯 Practical tip
Test with a question out of scope and confirm that the agent doesn’t make things up — it should fall back. This “question with no source” test is the best proof that your RAG is honest.
Key concepts
High / medium / low.
Parse the score.
Answer or use a fallback.
"I don't know" isn't a bug.
Wrapping up the module: what a good RAG agent does when the search doesn’t find nothing relevant?
📌 Module Summary
Next Module:
Track 3 — Prompts & Conversation Patterns