Measuring Vector Databases and Embeddings: KPIs That Prove Value

An embedding model, such as those from OpenAI, Google, Cohere or open-source options like the sentence-transformers family, turns a piece of text into a list of numbers. Texts with similar meaning end up close together. The vector database then answers questions like “which ten help articles are closest to this customer question?” in milliseconds.

Two choices drive quality: the embedding model and how content is split into chunks before embedding. A model that handles Turkish poorly will retrieve weak matches for Turkish queries. Chunks that are too long mix several topics; chunks that are too short lose context. Neither problem shows up as an error message. Only measurement reveals it.

Build a test set first: a list of real queries, each paired with the documents or products that a human agrees are correct answers. Fifty to two hundred queries is a reasonable start for most marketing use cases. Then measure the system against it.

• Recall at k: share of queries where a correct item appears in the top k results, often k equals 5 or 10.
• Mean reciprocal rank: how high the first correct result appears, on average.
• Precision at k: share of the top k results that are actually relevant.
• Latency at the 95th percentile: how long slow queries take, not just the average.
• Cost per thousand queries: embedding calls, database usage and any reranking model.

Rerun the test set every time you change the embedding model, chunk size or index settings. A change that looks small in configuration can move recall noticeably in either direction.

Approximate search trade-offs

Most vector databases use approximate nearest neighbour indexes such as HNSW, which trade a little accuracy for large speed gains. Settings like ef_search in HNSW control that balance. Measure recall and latency together when tuning; maximising one alone usually hurts the other.

Technical metrics convince engineers. Budgets are approved on business metrics. Pick the ones that match the use case.

• Site search: search exit rate, zero-result rate, conversion rate of sessions that use search, revenue per search session.
• RAG assistant: resolution rate, share of answers with a correct cited source, handoff rate, repeat contact rate.
• Recommendations: click-through on recommended items, add-to-cart from recommendations, revenue per visitor against a holdout.
• Content discovery: pages per session from related-content modules, return visits.

Zero-result rate is a quick win to watch. Keyword search often returns nothing for synonyms or typos. Semantic search should reduce that sharply. If it does not, check language support and whether product data was embedded at all.

Use two layers. The engineering layer tracks recall, latency, error rate and cost, ideally from the database’s own monitoring plus a scheduled run of the test set. The business layer, in GA4 and Looker Studio or your BI tool, tracks the outcome metrics above.

Connect them with a shared event. For site search, send a GA4 event with the query, the number of results and the search type, semantic or keyword. That lets you compare outcomes by search type and spot queries where semantic results underperform.

Testing only on easy queries is the most common. If the test set was written by the person who built the system, it tends to mirror the content’s own wording. Pull real queries from search logs and support tickets instead, including misspellings and mixed languages.

The second is ignoring freshness. When products go out of stock or articles are updated, embeddings must be refreshed. Track index lag: the time between a content change and its updated vector. A recommendation module suggesting discontinued products hurts trust quickly.

The third is skipping a keyword baseline. Hybrid search, which combines keyword matching such as BM25 with vector similarity, often beats pure vector search for product names and codes. Measure both before deciding.

Costs come from three places: generating embeddings when content is added or changed, storing vectors, and running queries. Managed services such as Pinecone price by usage or capacity; self-hosted options like Qdrant or pgvector shift cost to your own servers and staff time. At small scale, pgvector inside an existing PostgreSQL database is often the cheapest route.

Embedding dimension affects storage and speed. Some current models let you shorten embeddings with limited quality loss. Test a smaller dimension against your test set before paying to store full-size vectors for millions of items.