The most common retrieval-augmented generation (RAG) mistakes are feeding the system messy source content, splitting documents badly, trusting retrieval without testing it, and launching without a way to measure answer quality. Most RAG failures look like model problems but start in the data and retrieval layers. Agencies that build reliable RAG assistants spend more time on content preparation and evaluation than on choosing the language model.
What is retrieval-augmented generation?
RAG is a way of making a language model answer from your own information. When a user asks a question, the system searches a knowledge base, usually with embeddings stored in a vector database, retrieves the most relevant passages, and gives them to the model with instructions to answer from them. The model writes the answer; the retrieval step decides what it knows.
Common tools include frameworks such as LangChain and LlamaIndex, vector databases such as Pinecone, Weaviate, Qdrant or pgvector, and managed options such as OpenAI’s file search, Google Vertex AI Search and Amazon Bedrock Knowledge Bases. The mistakes below apply whichever stack you choose.
Mistake 1: Indexing content nobody has cleaned
Teams often point a RAG system at a whole shared drive or website and expect good answers. The result is a knowledge base full of old price lists, draft documents, duplicate pages and contradictions. The model retrieves whatever matches best, even if it is out of date.
- Choose sources deliberately and list them, with an owner for each.
- Remove drafts, superseded versions and duplicates before indexing.
- Resolve contradictions, such as two different return policies, at the source.
- Add metadata like date, product and region so retrieval can filter.
Mistake 2: Chunking documents badly
Documents are split into chunks before being embedded. Chunks that are too small lose context; a sentence about “the fee” means nothing without the paragraph that names the service. Chunks that are too large mix several topics and dilute retrieval.
What works better
Split along the document’s own structure: headings, sections and list items. Keep a heading with its content. Add a little overlap between chunks, and store the document title and section path with each chunk. Test several chunk sizes on real questions rather than picking a number from a tutorial.
Mistake 3: Relying on vector search alone
Semantic search finds meaning but can miss exact terms such as product codes, names and error numbers. A user asking about model “X200-B” may get passages about a similar product.
Hybrid search combines keyword search, such as BM25, with vector search, and is now supported by most vector databases and search platforms. Adding a reranking step, where a separate model reorders the top results by relevance, often improves answers further. Both are low-effort changes with clear benefits in many projects.
Mistake 4: Weak instructions to the model
If the prompt does not tell the model to answer only from retrieved content, it will fill gaps with general knowledge, which may be wrong for your business. If it does not say what to do when the answer is missing, it will guess.
Instruct the model to cite which passages it used, to say clearly when the information is not available, and to suggest contacting a person in those cases. Keep tone and format rules in the same system prompt so every answer looks consistent.
Mistake 5: No evaluation set
Many teams test a RAG assistant by asking it a few questions they thought of on the day. That misses the hard cases. Without a fixed test set, nobody can tell whether a change helped or hurt.
- Collect fifty to two hundred real questions from support tickets, sales emails or search logs.
- Write the correct answer and the source document for each.
- Score retrieval separately: did the right passage appear in the top results?
- Score answers on correctness, completeness and whether they stayed within the sources.
- Rerun the full set after every change to content, chunking, prompts or models.
Frameworks such as Ragas and tools inside LangSmith, Langfuse or vendor platforms can automate parts of this, including model-based scoring. Keep a human review of a sample, because automated judges also make mistakes.
Mistake 6: Ignoring permissions and privacy
An internal assistant that retrieves from every document can reveal salaries, contracts or client data to anyone who asks. Retrieval must respect the same access rights as the original files. Customer-facing assistants should never index personal data or internal notes.
Check data processing terms for each service in the pipeline, and follow regulations such as GDPR or KVKK where personal data is involved.
Mistake 7: Launching and forgetting
Content changes. Prices update, products launch, policies are rewritten. If the index is not refreshed, the assistant drifts out of date while still sounding confident. Schedule re-indexing, track which documents changed, and review a sample of live conversations every week, especially the ones where users asked for a person or rated the answer poorly.
How Alien Road applies retrieval-augmented generation
In a typical engagement Alien Road begins with a source audit: which documents and pages the assistant should know, who owns them, and where they conflict. We clean and restructure priority content first, often rewriting long pages into single-topic sections that both people and retrieval handle better.
Setup follows the practices above: structure-based chunking, hybrid search with reranking where the platform allows, clear system instructions and permission-aware retrieval. Before launch we build an evaluation set from the client’s real questions and agree a quality threshold.
After launch, we report retrieval accuracy, answer quality scores, handoff rates and the top unanswered questions each month. Unanswered questions become content tasks for the client’s team, which is often where the biggest gains come from.
Where to start this week
If you already run a RAG assistant, collect fifty real questions it has received and check, for each, whether the right document was retrieved. If fewer than most were, fix content and chunking before touching the model. If you are planning one, list your sources and owners first; that list will decide the project’s quality more than any tool choice.
Keep reading: AI Optimization · AI Optimization