RAG in production: what changes after the demo

A retrieval-augmented generation demo takes an afternoon: chunk some documents, embed them, retrieve the top five and ask a model to answer. Production is different, because real users ask questions the demo never saw, and wrong answers have consequences.

Build the evaluation set first

Collect a few hundred real questions — from tickets, search logs or interviews — with the answer a good employee would give and the source it comes from. Every change to chunking, retrieval, prompts or models gets scored against this set before release. Without it, you are tuning by vibes.

Hybrid retrieval beats pure vectors

Product names, SKUs, error codes and policy numbers are exactly where embeddings struggle. Combine keyword search with vector search, then re-rank. It is less elegant and far more accurate.

Permissions are part of retrieval

If an employee can't open a document, the assistant must not quote it. Store access control with each chunk and filter at query time, per user. Retrofitting this later is painful.

Freshness and ownership

Sync sources on a schedule, track when each chunk was last updated, and surface stale or conflicting content to the people who own it. RAG quality is mostly content quality.

Fail honestly

When retrieval returns nothing relevant, the right answer is "I don't know — here's who to ask". Instruct for it, test for it, and measure how often it happens. Those gaps are your documentation roadmap.

Watch cost and latency

Cache embeddings and frequent answers, route simple questions to smaller models, and set token budgets per request. Latency matters more than most teams expect for adoption.