RAG & Vector Databases
2 min read
Retrieval-Augmented Generation (RAG)
LLMs are frozen at training time and hallucinate on facts they never saw. RAG fixes this by retrieving relevant documents at query time and feeding them to the model as context.
The problem RAG solves
- The model doesn't know your private/internal data.
- Its knowledge is stale (fixed cutoff).
- You need citations and grounded answers.
The two phases
1. Indexing (offline)
Docs ─▶ Chunk ─▶ Embed (embedding model) ─▶ Vector DB
2. Retrieval + generation (online)
Query ─▶ Embed ─▶ Vector DB (similarity search) ─▶ top-k chunks
└─▶ Prompt = system + chunks + query ─▶ LLM ─▶ grounded answer
Vector databases
A vector DB (Pinecone, Weaviate, pgvector, Milvus) stores embeddings and finds the nearest ones fast using Approximate Nearest Neighbour (ANN) indexes (e.g. HNSW). Similarity is usually cosine distance.
Design considerations
| Decision | Trade-off |
|---|---|
| Chunk size | Small = precise but fragmented; large = context but noisy |
| top-k | More chunks = more context but more cost + distraction |
| Hybrid search | Vector + keyword (BM25) for names/IDs vectors miss |
| Re-ranking | A second model reorders candidates for relevance |
Common failure modes
- Bad retrieval → bad answer. Retrieval quality dominates output quality.
- Stale index — re-embed when source docs change.
Takeaway: RAG is a search problem wearing an LLM hat. Invest most effort in the retrieval pipeline, not the prompt.