What is RAG: how it works and why it decides what AI cites

RAG (retrieval-augmented generation) is the technique that makes a language model look up information in external sources before answering, instead of replying only with what it learned in training. If you've ever asked ChatGPT something and the answer came with three links next to it (and none of them was yours), you've already seen a RAG system at work.

Most guides on RAG are written for whoever is going to build a chatbot with their company's documents, and that's fine, because that's where the technique was born. But there's a second use that matters a lot more to a marketing team: ChatGPT, Perplexity and Google do retrieval-augmented generation over the web every time they answer with sources. My take is simple: in those search engines, your brand's visibility is decided first at the retrieval step. If your page doesn't show up in what the AI searches for, or doesn't have a passage that makes sense on its own, the LLM never reads it.

First how the technique works, then what changes for your site.

What is RAG?

RAG stands for retrieval-augmented generation, an artificial intelligence technique. The name describes pretty well what happens: the system retrieves information from a knowledge base, adds it to the question (that's the "augmented" part) and only then does the model generate the answer.

The acronym was coined in 2020 by a Facebook AI Research team (now Meta) led by Patrick Lewis, in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The underlying idea was to solve something any LLM user has already noticed: these models know a lot, but they have no information about what happened yesterday or about what your company's internal manual says.

The analogy I like most is an open-book exam. A model without RAG answers from memory, with its training data, and when it can't remember, it improvises with a lot of confidence. With RAG it has the book open on the right page, and its job becomes reading carefully and answering with what's there.

Why does an LLM need RAG?

An LLM learns from a huge amount of text up to a cutoff date. After that date it knows nothing new, and it doesn't know your company's private data, today's catalog or the returns policy you changed on Tuesday either.

The problem is that models don't warn you when they don't know. They generate text that sounds plausible, and we call that a hallucination. AWS explains it with an overly enthusiastic new employee who answers every question with absolute confidence, even with no idea (we've all met someone like that).

The obvious alternatives have their limits. Pasting all the documentation into the prompt works with little material, but models have a limited context window and, with too much information, the answer comes out slower, more expensive and less precise. Retraining with your data doesn't scale either, because you'd have to repeat it every time something changes. RAG solves this by handing the model the right information at the moment of answering, without touching the training data.

How does RAG work step by step?

A RAG system works in two phases. The first, indexing, happens before anyone asks anything. The second happens every time a query comes in.

Phase 1: preparing the knowledge base

1. Load the documents. This is the external material the model doesn't come with: PDFs, product sheets, web pages, support tickets or databases.
2. Split them into chunks. Retrieving an entire book isn't very useful, so the content is cut into chunks of a few hundred words. If a chunk is too big, it mixes topics; if it's too small, it loses its meaning.
3. Turn each chunk into numbers. An embedding is a list of numbers that represents the meaning of a text, and those vectors make it possible to find texts that say the same thing with different words.
4. Store them in a vector database, such as Pinecone or ChromaDB, which holds the embeddings and lets you search for the most similar ones.

Phase 2: answering a question

1. The user's question is turned into a vector with the same embedding model.
2. The retriever looks for the relevant information, which means the closest chunks. It's a semantic search, by meaning. Many systems combine it with a keyword search, because proper names, product codes and numbers get lost easily when you only search by meaning.
3. The results are reordered (re-ranking) and the best ones are kept, for example the five most useful.
4. An augmented prompt is built with the original question, what was retrieved (the context) and an instruction like "answer only with this information and cite the source".
5. The LLM generates the answer and, if the system is designed for it, delivers it with citations.

All of this happens in a few seconds. And notice a detail that comes back later: the LLM only sees what won in step 2. If something wasn't retrieved, the model answers without it.

What are the components of a retrieval-augmented generation system?

IBM sorts it into four pieces:

• The knowledge base: the repository with the documents already chunked and turned into vectors.
• The retriever: the component that looks for the relevant information for each question.
• The integration layer: coordinates the flow and builds the prompt the generator receives.
• The generator: the LLM that writes the final answer, such as GPT, Claude or Llama.

What are the benefits of RAG?

• No need to retrain. Updating documents is much cheaper than retraining.
• Current information. The model answers with the data the source has today, even if its training ended a year ago.
• Answers with citations. The user can go to the source and check whether what it said holds up, which improves trust in the tool.
• Fewer hallucinations. By anchoring answers in real documents, models make up less and gain precision (less doesn't mean zero either, we'll get to that).
• Control. Whoever runs the system decides which sources go in, which go out and which user can see each document. Confidential data stays in a separate base and doesn't get mixed into training.

What is RAG used for in companies?

The most common uses in the IBM and AWS guides and in the projects I see:

• Customer service: an assistant that answers each user with the current manuals and FAQs.
• Ecommerce: checking stock, features or up-to-date prices in the catalog databases before recommending a product.
• Banking, insurance and legal: answering with the current credit policy, insurance policy or regulation, where old data gets expensive.
• Internal search: helping a team find in seconds what's buried in the intranet documents.
• AI search engines: ChatGPT, Perplexity and Google use the same logic over the web. This case hits your site directly and I cover it in detail below.

What is the difference between RAG and fine-tuning?

Both techniques aim to make models work better on a specific topic, but they go about it differently:

They aren't mutually exclusive: fine-tuning adjusts the weights to teach the model how to behave, and RAG hands it the data of the day.

What are the limitations of RAG?

• Quality depends on what gets retrieved. If the retriever doesn't bring relevant information, or brings an old document or one with wrong data, the model answers with that and with the same confidence as always.
• RAG reduces hallucination, but doesn't eliminate it. Models can still mix chunks or connect figures that had nothing to do with each other.
• Badly cut chunks. A paragraph that depends on the previous one to make sense reaches the generator without context.
• Latency and cost. Every question adds a search, and keeping the index up to date is ongoing work.
• Privacy. If the knowledge base holds customer data, you need to control who can access which documents.

The first limitation is the one you notice most from the outside. In a study I did for Previsible with 500 prompts, about 78% of the content cited in commercial searches had been published or updated in the previous six months. Recency weighed a lot in that sample, so a page with 2023 data probably competes at a disadvantage, even if it was the best on its topic.

How do ChatGPT, Perplexity and Google use RAG?

An AI search engine is a retrieval-augmented generation system where the external data is the pages of the web and the knowledge base is an index. The scale changes, but the steps are the same: retrieve pages, extract passages, build the context with information from several sources and generate an answer with citations.

At Google, the Search Central documentation says AI Overviews and AI Mode may use a "query fan-out" technique, which issues several related searches across subtopics before answering. To appear as a link, a page has to be indexed and eligible to be shown with a snippet, and Google clarifies there are no additional technical requirements. AI Mode takes that logic further with many sub-searches in parallel.

In ChatGPT, search may share queries with providers such as Bing, and OpenAI uses its OAI-SearchBot crawler to surface sites in ChatGPT's answers; a site that blocks it is left out, according to its bots documentation. On top of that, ChatGPT now has its own search index. Perplexity, for its part, shows its sources next to what it answers.

Two Ahrefs studies put numbers on this. 76% of the pages cited in AI Overviews rank in Google's top 10 for that search, according to an analysis of 1.9 million citations. But only 12% of the URLs cited by ChatGPT, Gemini and Copilot appear in the top 10 for the original prompt (28.6% for Perplexity), according to another study with 15,000 prompts. My reading of that gap is fan-out: the system searches for variants of what you typed, sometimes in another language, as I explained in the article on ChatGPT fan-out queries.

What does RAG mean for SEO and a brand's visibility?

If the LLM only reads what was retrieved, the useful question for a brand is what its page needs to win that retrieval. This is what I would check:

1. That bots can get in and read the information. If your robots.txt blocks OAI-SearchBot, ChatGPT can't show you (I explained it in the robots.txt guide). And if the important content only appears after running JavaScript or inside accordions, several AI crawlers don't see it, because most read the HTML the server delivers.
2. Being indexed and ranking in the sub-searches. For AI Overviews, Google's index is still the way in. Ask yourself what the system would search for to build the answer (price, comparison, requirements, reviews) and whether you have a page that wins in each one.
3. Passages that make sense on their own. AI works with passages, so a section that says "as we saw above" arrives without context. In a directional analysis we did at Previsible on 5,000 AI responses, cited pages had one heading every 100 to 200 words, and the ones that barely appeared had one every 400 or more. 58% used headings phrased as questions.
4. Concrete, up-to-date data. Data with a date and a source is easy to extract and cite. A paragraph of generic opinion has thousands of similar competitors on the web.
5. Measure retrieval separately. Check whether you get cited, for which questions and with which page, using a fixed panel of prompts (I explain how in the share of voice guide).

None of these things guarantees a citation. What they do is take you off the list of pages the AI can't retrieve, which is already a lot.

Frequently asked questions

Are RAG and LLM the same thing?

No. The LLM is the model that generates text. RAG is the architecture that adds a retrieval step before generating. You can have an LLM without RAG, but not RAG without a language model.

Is RAG the same as semantic search?

Semantic search is one piece of RAG: the method used to find chunks by their meaning. RAG adds what comes next, which is passing that information to the LLM so it can write the answer.

Can you optimize a site for RAG?

You can make your content easy to retrieve and cite: accessible to bots, indexed, with clear sections and verifiable data. What you can't control is which sources the model picks in the end.

What I'd do this week

Take the five questions your customers ask before buying and type them into ChatGPT, Perplexity and Google. Note which pages get cited in each answer, with what data, and open those pages. Then open yours for the same question and look for the passage the AI could have used: if you can't find it in under ten seconds, the retriever probably couldn't either.