You measure retrieval-augmented generation on two levels. Technical quality covers whether the system retrieves the right documents and whether answers stay faithful to them. Business value covers whether users get correct answers faster, whether support tickets drop and what each answer costs. A dashboard should show both, because a fluent answer from the wrong document is still a wrong answer.
RAG is a pattern for building AI assistants that answer from your own content. When a user asks a question, the system searches a knowledge base, usually with vector embeddings and often keyword search too, retrieves the most relevant passages, and gives them to a language model to write an answer, ideally with citations. It is common in support bots, internal knowledge assistants and product search. Frameworks such as LangChain and LlamaIndex, and managed services from OpenAI, Google, AWS and others, make it easier to build.
Why do RAG projects need dedicated measurement?
A RAG system can fail in several places: documents may be missing or outdated, the search may retrieve the wrong passages, or the model may ignore the passages and improvise. From the outside, all of these look the same, a confident answer that is wrong.
Without measurement at each stage, teams try to fix the wrong part. They switch to a bigger model when the real problem was a stale help article, or rewrite prompts when search is returning irrelevant chunks.
Which retrieval KPIs matter?
Build a test set of 50 to 200 real questions, each with the document or passage that contains the correct answer. Use it to measure retrieval regularly.
- Recall at k: share of questions where the correct passage appears in the top k results, for example top 5.
- Precision at k: share of the top k results that are actually relevant.
- Mean reciprocal rank: how high the first relevant result appears on average.
- Coverage: share of real user questions for which a relevant document exists at all.
Coverage is the business metric hiding in the technical ones. If many questions have no matching document, the fix is writing content, not tuning search.
How do you measure answer quality?
Faithfulness and correctness
Faithfulness asks whether every claim in the answer is supported by the retrieved passages. Correctness asks whether the answer matches the known right answer. Both can be scored by human reviewers on a sample, and by an LLM-as-judge setup where a second model grades answers against the sources, which open-source tools such as Ragas support. Check automated grades against human grades before trusting them.
Citations and refusals
Track the share of answers with at least one citation, and the share where the system correctly says it does not know. A system that never refuses is probably inventing answers for questions outside its knowledge.
Which business KPIs prove value?
- Resolution or deflection rate: share of conversations solved without a human agent, confirmed by user feedback or no follow-up ticket.
- Escalation quality: when the bot hands over, does the agent get a useful summary?
- User feedback: thumbs up or down per answer, with optional comments.
- Time to answer for internal users, compared with searching documents manually.
- Cost per answered question: model tokens, embedding and vector database costs divided by answers.
- Latency: median and 95th percentile response time.
What should the dashboard look like?
Use three sections. The top shows business outcomes for managers: questions answered, deflection rate, satisfaction and cost per answer, with trends. The middle shows quality: faithfulness and correctness scores from the latest evaluation run, refusal rate and citation rate. The bottom shows operations: latency, error rate and the most frequent unanswered questions.
The unanswered questions list is often the most useful part. It tells the content team exactly which articles to write next.
How often should you evaluate?
Run the retrieval and answer test set after every significant change: new documents, a new embedding model, a different chunk size, a new language model or prompt. Also run it on a fixed schedule, monthly for example, because content changes over time.
Sample live conversations weekly for human review. Test sets drift away from what users actually ask; live samples keep you honest and give new questions to add to the set.
Which changes usually improve the numbers most?
Content fixes often beat model upgrades. Removing duplicate and outdated documents, splitting long pages into focused sections and adding missing answers tend to lift both retrieval recall and answer correctness.
On the technical side, hybrid search that combines keyword and vector matching helps with product names, codes and exact phrases, which embeddings alone can miss. A reranking step that reorders the top results before they reach the model is another common improvement. Measure each change on the test set before keeping it.
Keep a change log next to the dashboard, so every jump or drop in a score can be traced to a specific change.
How Alien Road applies RAG measurement
In a typical AI optimisation engagement, we start by auditing the knowledge base the RAG system will use: duplicates, outdated pages, missing topics. We then build the question test set with the client’s support or sales team from real tickets and chat logs.
During setup we measure retrieval and answer quality at each change, and after launch we run monthly evaluations and weekly live samples. The client receives a dashboard with business, quality and operations views, and a monthly list of content gaps drawn from unanswered questions.
Where to start this week
Export the last 200 questions customers asked your support team or chatbot. Group them by topic and, for each of the 30 most common, find the help article that answers it, or note that none exists.
That spreadsheet is the start of your test set and your coverage metric at once. Even before any RAG system is built, it shows how much of your content is ready to be used by one.
Share the list with the people who own the help content. Gaps found this way are concrete and easy to act on, which makes the case for the measurement work before any dashboard exists.
Keep reading: AI Optimization · AI Optimization