Retrieval-augmented generation, or RAG, has become the default architecture for enterprise AI. The premise is compelling: instead of relying on a language model's fallible internal memory, you retrieve relevant passages from your own trusted documents and give them to the model as context, so its answer is grounded in real sources. Done well, it turns a general-purpose model into a knowledgeable assistant for your policies, your products, or your knowledge base.
Done badly — which is common — it produces an assistant that sounds authoritative while quietly citing the wrong document, missing the critical caveat, or confidently answering a question the source material never addressed. Because the output looks grounded, these failures are harder to spot than a model that simply makes things up. The confidence is borrowed from the retrieval step; the error is introduced anyway.
RAG has two failure surfaces, not one
The mistake most teams make is treating a RAG system as a single black box and evaluating only the final answer. In reality it has two distinct stages, each of which can fail independently, and you cannot fix what you cannot separate.
The retrieval stage takes the user's question and finds the passages most likely to answer it. If retrieval fails — because the relevant passage was never indexed, the query was phrased differently from the source, the chunking split a crucial idea across two fragments, or the ranking buried the right passage beneath plausible-but-wrong ones — then the model is working from bad raw material. No amount of eloquence in the generation stage can rescue an answer built on the wrong sources.
The generation stage takes the retrieved passages and the question and composes an answer. Even with perfect retrieval, this stage can fail: the model may ignore the provided context and fall back on its training, blend the sources with invented details, misread a passage, or fail to notice that the retrieved material does not actually answer the question and should have produced "I don't know."
Because these two stages fail for different reasons and are fixed with different interventions, evaluating them separately is not a nicety — it is the only way to diagnose a RAG system that is underperforming. A system with 95% retrieval accuracy and a hallucinating generator needs completely different work than one with a faithful generator fed by poor retrieval.
The specific ways RAG breaks
Retrieval misses. The single most common cause of bad RAG answers is that the right information was never retrieved. Poor chunking is a frequent culprit: split documents too finely and you sever the context a passage needs to make sense; split too coarsely and you dilute relevance. Vocabulary mismatch is another — users ask in their words, documents are written in the organisation's words, and naive matching fails to bridge the gap.
Retrieval of the plausible-but-wrong. Sometimes retrieval returns passages that are topically related but answer a subtly different question — last year's policy instead of this year's, the consumer product's specification instead of the enterprise one. The model, trusting its context, produces a fluent answer to the wrong question.
Context ignored. A well-documented failure is the model that has the correct passage in front of it and answers from its training data anyway, especially when its internal "belief" conflicts with the source. The whole point of RAG — grounding — silently fails.
No graceful "I don't know." Many RAG systems will always produce an answer, even when the retrieved material is irrelevant or the knowledge base simply does not cover the question. A system that cannot say "the documents do not address this" will confidently fabricate, and users have no way to tell a grounded answer from an ungrounded one.
Staleness and drift. The knowledge base is a living thing. Documents are updated, superseded, or removed, but the index lags. A RAG system answering from an outdated snapshot delivers yesterday's policy with today's confidence.
How to evaluate RAG honestly
Serious RAG evaluation measures each stage against a curated test set of representative questions with known correct answers and known correct source passages.
For retrieval, the question is whether the right passages were fetched. Metrics here ask: of the passages that should have been retrieved, how many were (recall)? Of those retrieved, how many were actually relevant (precision)? Was the best passage ranked near the top, or buried? Retrieval evaluation can and should be automated against a labelled set, and it is where the largest, cheapest improvements usually hide.
For generation, the central question is *faithfulness*: does the answer follow from the retrieved passages, or does it add, contradict, or distort them? Every factual claim in the answer should be traceable to a retrieved source. A separate question is *relevance*: does the answer actually address what the user asked? And critically, *appropriate refusal*: when the sources do not support an answer, does the system decline rather than invent?
Crucially, these evaluations must run on questions that resemble real usage — including the ambiguous, the adversarial, and the ones the knowledge base cannot answer — not a tidy set of questions you already know the system handles well. A RAG system that scores beautifully on softball questions and collapses on realistic ones has been evaluated for reassurance, not for truth.
And like all AI evaluation, this is not a one-off. Every change to the documents, the chunking strategy, the retrieval configuration, or the underlying model can shift behaviour. Regression testing against a stable benchmark catches the improvement that quietly broke something else.
The bottom line
RAG feels safe because the answers appear to come from your own trusted documents. That appearance is exactly what makes its failures dangerous: a confident, well-cited, wrong answer is more persuasive than an obvious hallucination. The organisations that get real value from RAG are the ones that stop treating it as a black box, evaluate retrieval and generation separately, test on realistic and unanswerable questions alike, and monitor for the drift that turns a trustworthy assistant into a confident liar over time.