Every "chat with your documents" product you have seen is built on the same idea. It has a name, retrieval-augmented generation, and it is far less mysterious than the acronym suggests.
I build these for businesses, and the most useful thing I can do is explain it without the jargon, so you can tell when it is the right tool and when someone is selling you complexity.
The problem RAG solves
A language model knows a great deal about the world in general and nothing about your business in particular. It has never read your returns policy, your price list, your onboarding manual or last quarter's contract terms.
Ask it about them anyway and it does something unhelpful: it produces a fluent, confident, plausible answer built from how businesses like yours usually work. That answer can be wrong in ways that are hard to spot, because it sounds right.
RAG fixes this by changing the question. Instead of asking the model to remember your documents, you look up the relevant passages first and hand them over with the question. The model's job shrinks from "know the answer" to "read this and answer from it", which it is much better at.
The simplest way to think about it: an open-book exam instead of a memory test. The model is not smarter. It is just allowed to look at the right page.
How it works, in five steps
- Split your documents into chunks. A long manual becomes hundreds of short passages, each small enough to be useful on its own.
- Turn each chunk into an embedding. An embedding is a list of numbers that captures what a passage is about, so passages with similar meaning end up numerically close together.
- Store them in a vector database. This is a database built to answer the question "which stored passages are closest in meaning to this one?" quickly.
- When someone asks a question, embed the question too and fetch the handful of closest passages.
- Send the question plus those passages to the model, with an instruction to answer only from what it was given.
Steps one to three happen once, when you load your documents, and again whenever they change. Steps four and five happen on every question.
What you actually need
| Piece | Job | Common choices |
|---|---|---|
| Your documents | The source of truth | PDFs, Google Docs, help pages, spreadsheets |
| Embedding model | Turns text into searchable numbers | Offered by every major model provider |
| Vector store | Finds the closest passages | Supabase with pgvector, Pinecone, others |
| Language model | Writes the answer from the passages | Claude, GPT-class, Gemini |
| Orchestrator | Wires the steps together | n8n, Make.com, or custom code |
If you already use Supabase, the vector store is a feature you switch on rather than a new service to run. That keeps the moving parts to a minimum, which matters more than people expect. I wrote about choosing the surrounding tools in the AI tool stack every business should know.
Where RAG quietly fails
The demos always work. The failures show up in week three, and they are almost never about the AI.
Bad source documents
If your returns policy contradicts itself across two pages, the assistant will cheerfully quote either one. Building a RAG system usually exposes gaps and contradictions in your own documentation. That is a benefit, but only if you fix them.
Poor chunking
Cut a document in the wrong place and the answer ends up split across two chunks, with the retriever fetching only one. A table separated from its heading is a classic example. Chunk along natural boundaries such as headings and sections, not at a fixed character count.
Retrieval that misses
Sometimes the right passage exists and the search simply does not find it, because the customer used different words from the document. When that happens the model has nothing relevant to read, and a badly instructed model will answer anyway.
Stale content
A policy changed in March and the old version is still loaded. The assistant is now confidently out of date. Whoever owns the policy should own the document the assistant reads, with a re-index whenever it changes.
The rule that prevents most disasters: instruct the model to answer only from the retrieved passages, to quote or cite them, and to say "I do not have that information" when nothing relevant came back. Then give it somewhere to send the question, such as a human. An assistant with no way to say "I do not know" will invent an answer instead.
When you do not need it
- Your material fits in one prompt. A one-page FAQ can simply be pasted into the instructions. Retrieval adds machinery you then have to maintain.
- The answer is a lookup, not a reading task. "Where is order 4821?" needs a database query, not a vector search.
- The documents are badly organised and nobody will fix them. RAG will faithfully surface the mess.
How to start small
- Pick one narrow, well-documented area, such as returns and shipping.
- Clean those documents first. Remove duplicates and resolve contradictions.
- Build the five steps above, with the answer-only-from-sources instruction.
- Test with real questions from real customers, including the vague and badly spelled ones.
- Run it in shadow mode, where it drafts and a human sends, before letting it answer alone.
That staged approach is the same one I recommend in how to automate customer support with AI safely, and the reasoning about when an assistant should decide for itself is in AI agents vs AI automation.
Done well, RAG is not impressive technology. It is a careful way of making sure the AI reads the right page before it speaks, and that is exactly why it works.