RAG Evaluation: How to Know Your AI Answers Are Right

Retrieval-augmented generation is often sold as the cure for hallucination: the model answers from your documents, so it cannot make things up. The evidence says otherwise. In 2024, Stanford RegLab and the Stanford Institute for Human-Centered AI benchmarked AI legal research tools whose vendors promoted retrieval as a safeguard, and in one case advertised hallucination-free citations. Lexis+ AI and Ask Practical Law AI produced incorrect information more than 17% of the time, and Westlaw AI-Assisted Research hallucinated more than 34% of the time.

The study's definition is the useful part for any business team. A response could be incorrect, meaning it stated something false, or misgrounded, meaning the statement was right but the cited source did not support it. Both are failures for a system people are meant to trust, and neither shows up when a team tries ten friendly questions in a demo. Measuring them takes a test set, a scoring method and thresholds agreed in advance.

A RAG answer passes through two steps that fail in different ways. Retrieval can miss the passage that holds the answer, or bury it under irrelevant chunks. Generation can then ignore good context, add facts from the model's own memory, or leave out part of what the sources say. A single overall score hides which step broke, so the fix becomes guesswork.

Microsoft's Foundry documentation makes the split explicit. It calls retrieval checks process evaluation and answer checks system evaluation. It also offers a compact way to think about the answer: groundedness measures the precision side, meaning the response contains nothing outside the grounding context, while response completeness measures the recall side, meaning it does not miss critical information from the expected answer. You need both. A short answer can be perfectly grounded and still useless.

OpenAI's evaluation guide recommends mixing data sources, including production logs, historical data and human-curated examples, and warns against test data that does not reproduce real traffic patterns. Anthropic's guidance adds that more questions with slightly noisier automated grading beat a handful of perfectly hand-graded ones. For a company assistant, the best raw material is usually already on file: support tickets, sales questions, onboarding queries and the questions employees ask each other in chat.

For each question, record a reference answer written or approved by someone who owns the content, and the document or section that should be retrieved. Those two fields unlock the metrics that need ground truth. Then make sure the set covers the cases that break RAG systems:

Two metrics from the open-source Ragas library cover most retrieval questions. Context recall measures how much of the needed information was retrieved: Ragas splits the reference answer into claims and calculates the share that the retrieved context supports. It therefore always needs a reference. Context precision measures whether relevant chunks are ranked above irrelevant ones, as a mean of precision at each rank. Low recall means the answer was never available to the model; low precision means it was there but diluted by noise.

If you label which documents are relevant to each question, ranking metrics such as NDCG become available. Microsoft recommends using them in a parameter sweep: generate retrieval results for different search methods (vector or semantic), top-k values and chunk sizes, then keep the configuration that scores best. This is far cheaper than tuning prompts while the real problem sits in the index.

Faithfulness is the core answer metric. Ragas identifies every claim in the response and checks whether each one can be inferred from the retrieved context; the score is the supported claims divided by all claims, from 0 to 1. An answer with two claims, one supported, scores 0.5. Pair it with a completeness check against the reference answer, and with a relevance check on whether the response addresses the question asked.

The combination tells you where to look when a test fails:

Most of these metrics rely on a model grading another model's output, because reading thousands of answers by hand is too slow. The approach has research behind it: the 2023 MT-Bench and Chatbot Arena study found that strong LLM judges reached over 80% agreement with human preferences, the same level humans reach with each other. The same paper documented position, verbosity and self-enhancement biases.

The vendors' own guidance turns that into practice. OpenAI says pass/fail or pairwise comparison is more reliable than open-ended scores, that judges should be calibrated against human feedback before scaling up, and that response length should be controlled for. Anthropic recommends detailed rubrics, a fixed output such as correct or incorrect, letting the grader reason before scoring, and grading with a different model from the one that produced the answer. In practice, have a content owner label a sample of answers, compare the judge's verdicts with theirs, and only trust the judge once the two agree.

A score only matters if it decides something. OpenAI's guide gives an example target for question answering over documents: context recall of at least 0.85, context precision above 0.7 and at least 70% of answers rated positively. Treat numbers like these as a starting point to adjust for your risk, and write your own before the first test run so the result cannot move the goalposts. A tool that answers HR policy questions deserves a stricter faithfulness bar than one that drafts internal summaries.

OpenAI also recommends running evaluations on every change, logging everything and growing the test set over time. For a RAG system, that becomes a short checklist: