What we learned building production RAG systems for enterprise clients

A RAG demo is easy: embed some documents, retrieve the top matches, stuff them into a prompt. Production RAG is a different discipline entirely, and most of the hard problems live outside the model.

Chunking strategy matters more than model choice. Poorly chunked documents, cut mid-sentence, missing headers, no metadata, produce confidently wrong answers regardless of which LLM sits downstream. We spend a disproportionate amount of early engagement time on document structure and metadata before touching the retrieval pipeline.

Evaluation has to be built before the feature ships, not after users complain. We build a small labeled evaluation set from real queries early in every project, and re-run it against every prompt or retrieval change so regressions are caught before a customer finds them.

Pure vector similarity search misses queries that hinge on an exact term, a product code, or a name the embedding model doesn't weight heavily. Hybrid search, combining vector similarity with keyword/BM25 matching and re-ranking the merged results, consistently outperforms either approach alone once the document set is large or the queries get specific. It costs more to build and tune, and it's worth it for anything beyond a small, homogenous knowledge base.

Metadata filtering is what makes retrieval trustworthy at scale, not just accurate. Tagging chunks with source, date, access level, and document type lets you constrain retrieval before similarity search even runs, which matters both for answer quality (don't retrieve a superseded policy document) and for access control (don't retrieve a document this user isn't supposed to see, regardless of how well it matches the query).

Guardrails and confidence signals matter as much as accuracy. A system that says "I'm not sure, here's a human to ask" outperforms a system that answers confidently and wrong, especially in support and compliance-adjacent use cases where a wrong answer costs more than a slow one.

Observability has to be built in from day one, not added when something breaks. We log the retrieved chunks alongside the generated answer for every production query, not just the final output, because when an answer is wrong, the fix is almost always in retrieval, and you can't debug what you didn't capture.

Cost and latency are product decisions, not infrastructure afterthoughts. Which model handles which query, when to cache, and when a smaller model is good enough are decisions we make explicitly, because they change the unit economics of the feature.