Building a Production RAG Pipeline: What I Learned the Hard Way

RAG looks deceptively simple on paper. Embed a document corpus, store vectors, embed the query, retrieve the top-k chunks, shove them into an LLM prompt, and get an answer. Every tutorial stops there. Production doesn't.

I built two RAG systems — a Research-grounded Decision Intelligence Engine and an internal RAG Eval Harness — and both times I hit a wall of problems that nobody had blogged about. This is what I wish I'd read first.

1. Chunking is not 'just split by 512 tokens'

The first mistake I made was using fixed-size token chunking with no overlap. The result: retrieved chunks that started mid-sentence and ended mid-thought, giving the LLM context that was technically present but semantically useless.

What actually works depends on your document type. For dense technical papers I moved to semantic chunking — splitting on paragraph boundaries and measuring cosine similarity between adjacent chunks. If similarity drops sharply, that's a natural break. For structured documents with headers, recursive character splitting with a 200-token overlap between chunks gave much better results.

2. Retrieval quality is the real bottleneck — not the LLM

I spent two days tuning prompts when the actual problem was that I was retrieving the wrong chunks. The LLM can only answer well if what you retrieved is actually relevant. Garbage in, garbage out — no matter how good your model is.

Pure vector search (cosine similarity on embeddings) is great for semantic similarity but terrible at exact keyword matching. Someone asking 'what is the CUSUM threshold in the paper?' needs a keyword hit, not a semantically adjacent paragraph about statistical process control.

The fix: hybrid search. BM25 for keyword recall, dense vectors for semantic recall, then a reciprocal rank fusion to merge the two result sets. In my system this alone improved context recall from 0.61 to 0.84 on the RAGAS eval suite.

# Hybrid search: merge BM25 and vector results via RRF
def reciprocal_rank_fusion(bm25_results, vector_results, k=60):
    scores = {}
    for rank, doc in enumerate(bm25_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
    for rank, doc in enumerate(vector_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
    return sorted(scores.keys(), key=lambda d: scores[d], reverse=True)

3. You need a re-ranker

Even with hybrid search, the top-5 retrieved chunks aren't always the most relevant 5. Embedding models optimise for approximate nearest-neighbour search — they're fast but not precise. A cross-encoder re-ranker reads the query and each candidate chunk together and gives a relevance score. It's slower (runs the model once per chunk), but you only run it on your top-20 candidates.

I used a sentence-transformers cross-encoder (ms-marco-MiniLM-L-6-v2) as a re-ranking step before passing context to the LLM. Faithfulness scores on RAGAS went from 0.79 to 0.91. That's the difference between a RAG system that halluccinates occasionally and one that's actually reliable.

4. No observability = flying blind

The scariest thing about a broken RAG system is that it doesn't crash. It just gives confident-sounding wrong answers. Without observability you have no idea which queries are failing, which chunks are being retrieved, or whether a model update broke your pipeline.

• —Log every query, retrieved chunk IDs, and the final answer
• —Track RAGAS metrics (faithfulness, context recall, answer relevance) on a sample of queries continuously
• —Alert when faithfulness drops below your baseline — this usually means a retrieval regression, not an LLM problem
• —Store query embeddings over time to detect distribution shift (new question types you weren't indexed for)

In the RAG Eval Harness I built a GitHub Actions CI gate that runs the RAGAS metric grid on every PR that touches chunking, embedder config, or retrieval params. If faithfulness drops more than 5% from baseline, the PR fails. This caught 3 regressions during tuning that would have shipped undetected.

5. The metric that actually matters

Everyone reports answer quality metrics. Almost nobody reports the metric that matters most in production: latency at the 95th percentile. A RAG system that's accurate but returns answers in 8 seconds will get abandoned. My target was sub-800ms p95 end-to-end.

The biggest latency wins came from: async retrieval (fire the BM25 and vector search in parallel), caching embeddings for repeated queries in Redis, and keeping the LLM context window tight — retrieving 3 highly relevant chunks beats retrieving 10 mediocre ones, and it's faster.