Retrieval-augmented generation (RAG) costs come from four places: preparing your documents, storing them as vector embeddings, running retrieval on every question, and paying a language model to write the answer. For most marketing and support use cases, the model tokens and the people who maintain the content cost far more than the database. ROI depends on how many repetitive questions the system answers correctly without a human.
What is RAG, in plain terms?
RAG is a pattern, not a product. A user asks a question, the system searches a private knowledge base for the most relevant passages, and those passages are handed to a large language model as context. The model then answers using that material instead of relying only on what it learned during training.
The pattern exists because models do not know your price list, your return policy or last week’s campaign brief. Fine-tuning a model on that data is slow and goes stale. Retrieval keeps the facts outside the model, so updating an answer means updating a document.
Common building blocks at the time of writing include embedding models from OpenAI, Cohere or Google, vector stores such as Pinecone, Weaviate, Qdrant or pgvector inside PostgreSQL, and orchestration libraries like LangChain or LlamaIndex. Managed options also exist: OpenAI’s file search, Google Vertex AI Search and Amazon Bedrock Knowledge Bases bundle most of the stack.
Where does the money actually go?
Teams often budget for the vector database and forget everything around it. A realistic cost sheet has five lines, and the order below roughly matches how much attention each deserves.
- Generation tokens: every answer sends the question plus retrieved passages to a model. Long context windows make this the largest variable cost.
- Content preparation: cleaning PDFs, splitting documents into chunks, removing outdated pages and tagging metadata. This is mostly staff time.
- Embeddings: a one-time cost to embed the corpus, then small recurring costs for new or changed documents and for each incoming query.
- Vector storage and hosting: modest for a few thousand documents, more noticeable when you index millions of chunks or need low latency across regions.
- Evaluation and maintenance: test sets, monitoring dashboards and someone who reviews wrong answers every week.
Pricing for each of these changes often, so check current provider pages before you commit. What stays stable is the shape: generation and labour dominate, storage rarely decides the project.
Why chunk size changes the bill
If each retrieved chunk is 800 tokens and you pass five of them, every question carries about 4,000 tokens of context before the model writes a word. Halving chunk size or retrieving three passages instead of five cuts that cost directly. It can also hurt accuracy, which is why you test both settings against the same question set.
How do you build a first-year budget?
Start from volume, not from technology. Count how many questions the system will see per month, estimate average context size, and multiply by the model’s published per-token rate. Then add fixed costs.
- Step 1: Pull three months of support tickets, chat logs or sales enquiries and count repeat question types.
- Step 2: Pick a pilot scope, for example shipping and returns only, and estimate monthly query volume for that scope.
- Step 3: Price two models, a small fast one and a larger one, using your average prompt length.
- Step 4: Add setup labour for content cleanup, usually the single biggest one-off item.
- Step 5: Reserve a monthly maintenance block for reviewing failed answers and refreshing documents.
For example, an online furniture retailer answering delivery and assembly questions might find that a small model handles 90 percent of queries well, with the larger model reserved for escalations. That routing choice often matters more than any database decision.
Which metrics prove RAG is paying back?
ROI arguments fall apart when the only number is “questions answered”. A bot can answer thousands of questions badly. Track outcomes that a finance team already recognises.
- Deflection rate: share of conversations closed without a human agent, checked against repeat contacts within 48 hours.
- Answer accuracy: graded on a fixed test set of real questions, re-run after every content or model change.
- Cost per resolved query: total monthly RAG spend divided by correctly resolved conversations.
- Time saved for staff: minutes per ticket before and after, sampled rather than guessed.
- Revenue signals: for pre-sales assistants, assisted conversion rate compared with a holdout group.
Compare cost per resolved query with your current cost per human-handled ticket. If the gap is small, the project may still be worth it for speed or 24-hour coverage, but say so honestly in the business case.
What usually inflates RAG costs?
Three habits push budgets up. The first is indexing everything: old campaign decks, duplicate PDFs and draft pages add noise, increase retrieval mistakes and raise storage. The second is defaulting to the largest model for every question. The third is skipping evaluation, so nobody notices that a content change broke twenty answers until customers complain.
Caching helps. Many providers now offer prompt caching or cheaper rates for repeated context, and frequently asked questions can be served from a stored answer that is refreshed on a schedule.
How Alien Road applies RAG for clients
In a typical engagement we begin with discovery: we export real customer questions from the help desk, CRM or site search and group them by intent. That list becomes both the pilot scope and the evaluation set, so success is defined before anything is built.
Setup follows a narrow first release. We clean the source documents, agree chunking and metadata rules with the client’s content owner, and connect retrieval to a model that matches the budget. Measurement runs from day one: every answer is logged with the passages used, and a weekly review flags wrong or unsupported replies. Reporting goes to the client as a short monthly sheet with deflection, accuracy, cost per resolved query and the list of documents that need updating.
Where to start this week
Export last quarter’s most common 200 customer questions and write the correct answer for 50 of them. That small file is your test set, your scope document and your first ROI baseline. Price one small and one large model against it before choosing any vendor, and keep the spreadsheet: it will become the budget you defend later.
Keep reading: AI Optimization · AI Optimization