What Is RAG (Retrieval-Augmented Generation)? A Plain-English Guide for Businesses

Last updated: 5 October 2026

Short answer: retrieval-augmented generation (RAG) means an AI system first searches your documents for passages relevant to a question, then gives those passages to a language model and asks it to answer using them, ideally with citations. It lets an AI assistant answer from your policies, manuals or support tickets without retraining the model, and it keeps answers current when documents change. It reduces invented answers but does not eliminate them, so good systems show their sources and respect who is allowed to see what.

This guide is part of our practical guide to AI for small business.

Why do AI assistants need RAG?

A language model on its own only knows what was in its training data. It does not know your company's price list, last month's procedure change or the answer buried in ticket #4,217. Asked anyway, it may guess confidently.

RAG was introduced in a 2020 research paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, presented at the NeurIPS conference. Meta, whose researchers developed it, explained the key benefit: you can change what the system knows by swapping the documents, without retraining the model. That idea now sits behind most "chat with your documents" products.

How does RAG work, step by step?

Google's documentation for its Vertex AI RAG Engine describes the same stages most systems use:

1. Ingest. Collect the documents: PDFs, web pages, SharePoint files, support tickets.
2. Chunk. Split them into passages of a few hundred words, because models work better with focused excerpts than whole manuals.
3. Embed. Convert each chunk into an embedding, a list of numbers that captures its meaning, so that "annual leave" and "holiday allowance" end up close together.
4. Index. Store the embeddings in a vector database so they can be searched quickly.
5. Retrieve. When someone asks a question, convert the question the same way and find the most similar chunks, often combined with ordinary keyword search.
6. Generate. Send the question plus the retrieved chunks to the language model with an instruction such as "answer only from these sources and cite them".

The quality of step 5 decides almost everything. If the right passage is not retrieved, the model cannot use it.

A small example

An employee asks: "How many days can I carry over to next year?" The system finds three chunks from the HR policy that mention carry-over, sends them to the model, and gets back: "Up to five days, which must be used by 31 March (HR Policy, section 4.2)." The employee can click the citation and check.

RAG, fine-tuning or just a long prompt?

These three approaches are often confused:

The long-context option has become much more practical. As of September 2026, leading models from Anthropic, OpenAI and Google can accept around a million tokens in one request, well over 1,000 pages of text. Anthropic's own guidance says that if a knowledge base is smaller than about 200,000 tokens (around 500 pages), you can simply include the whole thing in the prompt, and prompt caching makes repeated questions much cheaper.

RAG still wins when:

• the collection is larger than a prompt can hold, or grows every day;
• permissions matter: a sales assistant should not quote from HR files the user cannot open;
• you want citations to specific passages;
• cost per question matters, because sending a thousand pages with every question is expensive even with caching.

Where does RAG go wrong?

• Retrieval misses. The right passage exists but is not found. Anthropic measured this directly: in its tests, a standard setup failed to retrieve the right chunk among the top 20 results 5.7% of the time. Adding context to each chunk, combining keyword and semantic search, and re-ranking the results cut failures by 67%, to 1.9%.
• Lost context. A chunk that says "the limit is 50 kg" is useless if the chunking separated it from the heading saying which product it refers to.
• Stale indexes. If documents change but the index is not refreshed, the assistant confidently cites the old version.
• Invented answers anyway. The model can still fill gaps or misread a passage. Citations let people catch this; without them, errors go unnoticed.
• Permission leaks. The security project OWASP lists vector and embedding weaknesses among its top ten risks for AI applications, including one user's query retrieving another department's confidential documents. Its advice is permission-aware retrieval and accepting data only from trusted sources.
• Prompt injection. A document can contain hidden instructions such as "ignore previous instructions". OWASP ranks prompt injection as the number one risk. Only index documents you trust.

What are the ready-made options?

You rarely need to build RAG from scratch now:

• Office suites. Microsoft 365 Copilot grounds answers in your organisation's email, files and meetings, with access scoped by each user's permissions. This is also why cleaning up over-shared SharePoint sites matters before rolling it out.
• Notebook tools. Google's NotebookLM answers from sources you upload, with in-line citations. It is good for a project folder or a set of standards.
• Cloud platforms. AWS Bedrock Knowledge Bases, Google Vertex AI RAG Engine and OpenAI's file search provide managed pipelines. Bedrock can filter by document access lists and include citations. OpenAI charges $0.10 per GB per day for file storage, with the first GB free.
• Open source. Frameworks such as LlamaIndex and LangChain, and vector search built into databases you may already run, such as pgvector for PostgreSQL. If your documents must stay in-house, these can run entirely on your own hardware, as in my guide to setting up a private AI assistant on your own PC.

Does it actually help? One published example

LinkedIn's customer service team published results of a system that combined RAG with a knowledge graph built from past support tickets. After about six months in use, median time to resolve each issue fell by 28.6%. That is a large company with a dedicated team, but the pattern, answering new questions from how similar ones were solved before, applies to any support desk.

How do you judge a RAG system?

Before trusting one with real work, test it:

1. Write 20 to 30 real questions with known answers, including a few whose answer is not in the documents.
2. Check faithfulness. Is every claim in the answer supported by the cited passage? Evaluation tools such as Ragas score this automatically as the share of claims backed by the retrieved text.
3. Check retrieval. Did it find the right passage? Wrong answers usually start here.
4. Check refusals. For questions the documents do not answer, it should say so rather than improvise.
5. Test permissions with two users who should see different documents.

Writing clear questions and instructions matters here too; the patterns in my practical guide to prompt writing apply. And if you are considering systems that act on the answers, not just report them, read what "agentic AI" really means first. In quality and engineering work, RAG is already changing how procedures and inspection records are searched, as I described in how AI is changing engineering documentation and quality work.

FAQ

Is RAG the same as training an AI on my data?
No. RAG leaves the model unchanged and looks up your documents at question time. Your documents can be updated or removed at any time without retraining.

Does RAG stop AI hallucinations?
It reduces them, because the model answers from real passages, but it does not stop them. Always prefer tools that show citations so answers can be checked.

Do I need a vector database?
For large collections, usually yes, although many managed services include one. For a few hundred pages, pasting the documents into a long-context model may be simpler.

Is my data safe in a RAG system?
That depends on where the documents and index are stored, who can query them, and whether retrieval respects each user's permissions. Check those three points before connecting sensitive files.