Retrieval-augmented generation (RAG) is a method where an AI model first searches your own documents for relevant passages, then writes an answer using those passages as context. Moving a RAG pilot into production needs a clean source inventory, sensible chunking, a tested retrieval layer, an evaluation set with known answers, access controls and monitoring. Skipping evaluation is the most common reason pilots look good in demos and fail with real users.
RAG became the default way to make language models answer from company knowledge without retraining them. A support assistant that reads the help center, a sales tool that quotes the current price list, an internal search over policy documents: all of these are RAG systems. The model stays general; the knowledge comes from retrieval at query time, so updating a document updates the answer.
What does a RAG system consist of?
Every RAG setup has the same basic parts, whatever framework you use.
- Sources: the documents, pages, product feeds or database tables you want the system to answer from.
- Ingestion: loading, cleaning and splitting those sources into chunks.
- Embeddings: converting chunks into vectors with an embedding model, so similar meanings sit close together.
- Index: a vector database or search engine, such as pgvector, Pinecone, Weaviate, Elasticsearch or OpenSearch, often combined with keyword search.
- Retriever: the logic that finds the best chunks for a question, sometimes followed by a reranker.
- Generator: the language model that writes the answer from the retrieved chunks.
- Application layer: the interface, logging, permissions and feedback.
Frameworks such as LangChain and LlamaIndex, and managed options in Azure AI Search, Google Vertex AI and Amazon Bedrock, package many of these pieces.
Phase 1: pilot checklist
The pilot should answer one question: can retrieval find the right information for real questions?
- Pick one narrow use case with a clear owner, such as answering shipping and returns questions.
- Collect 50 to 100 real questions from tickets, chats or search logs.
- Write the correct answer and the source document for each. This becomes your evaluation set.
- Inventory sources: list every document, its owner, update frequency and whether it is still accurate.
- Remove outdated, duplicate and contradictory documents before ingestion.
- Start with simple chunking, such as by heading or about a few hundred words with some overlap.
- Test retrieval alone: for each question, is the correct source in the top results?
- Only then connect the generator and test full answers.
How should you chunk and index content?
Chunking decides what the model can see. Chunks that are too small lose context; chunks that are too large dilute relevance and waste tokens.
Practical chunking rules
- Split on document structure first: headings, sections, FAQ pairs, table rows.
- Keep the document title and section heading attached to each chunk as metadata.
- Store the source URL, last updated date and access level with every chunk.
- Treat tables and product specs carefully; flattening a table into prose often breaks it.
Hybrid search, combining keyword matching with vector similarity, usually performs better than either alone, especially for product codes, names and exact phrases that embeddings handle poorly.
Phase 2: hardening for production
Once retrieval works, the checklist shifts to reliability and safety.
- Grounding instructions: tell the model to answer only from provided context and to say when the answer is not there.
- Citations: show the source link for every answer so users and reviewers can check it.
- Access control: filter retrieval by the user’s permissions, so a customer never sees internal-only documents.
- Prompt injection defense: treat retrieved text as data. A document containing “ignore previous instructions” must not change behavior.
- Personal data: exclude or mask sensitive fields at ingestion.
- Freshness: schedule re-ingestion when sources change, and delete chunks from removed documents.
- Fallbacks: route unanswered questions to a human or a contact form.
How do you know the system is good enough?
Measure retrieval and generation separately. If an answer is wrong, you need to know which part failed.
- Retrieval hit rate: share of evaluation questions where the correct source appears in the top results.
- Faithfulness: share of answers fully supported by the retrieved context.
- Answer correctness: share of answers matching the reference answer.
- Refusal accuracy: when the answer is not in the sources, does the system say so?
- Latency and cost per answer.
Tools such as Ragas, TruLens and the evaluation features in major cloud platforms can automate parts of this, but have humans review a sample every release. Set a target for each metric before launch and do not go live until you meet it.
Phase 3: monitoring after launch
- Log every question, retrieved chunk IDs and answer.
- Collect thumbs-up and thumbs-down feedback with an optional comment.
- Review unanswered and low-rated questions weekly; they reveal missing content.
- Re-run the evaluation set after any change to chunking, embedding model, prompt or generator model.
- Track which sources are retrieved most; make sure their owners know they are business-critical.
How Alien Road applies retrieval-augmented generation
In a typical engagement, discovery is mostly content work. We gather real questions from the client’s support inbox, site search and sales team, then map them to existing pages and documents. This usually shows gaps and contradictions that need fixing before any AI is involved.
Setup follows the checklist above: cleaned sources, structure-based chunking with metadata, hybrid search and a grounded prompt with citations. Permissions are designed with the client’s IT contact, and the evaluation set is agreed with the business owner so everyone uses the same definition of a correct answer.
Measurement uses the evaluation set at every release plus live feedback once the system is in use. Monthly reporting lists retrieval hit rate, faithfulness, unanswered question themes and the content updates they triggered. The same content improvements often help organic search too, because clearer pages answer questions better for people and search engines alike.
Where to start this week
Export the last 100 customer or employee questions from one channel. For each, find the page or document that contains the answer, and mark the ones where no good source exists. That spreadsheet is your evaluation set and your content gap list at the same time, and it is the foundation of any RAG pilot worth running.
Keep reading: AI Optimization · AI Optimization