Overview
RAG (Retrieval-Augmented Generation) enhances LLM responses by retrieving relevant context from a knowledge base. This guide covers production implementation.
RAG Pipeline
ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ
β Ingest ββββΊβ Chunk ββββΊβ Embed ββββΊβ Store ββββΊβ Index β
β Documentsβ β β β β β Vectors β β β
ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ
ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ
β Query ββββΊβ Embed ββββΊβ Search ββββΊβ Rerank ββββΊβ Generate β
β β β Query β β β β β β β
ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ
1. Document Ingestion
Loading Documents
from langchain_community.document_loaders import (
DirectoryLoader,
PyPDFLoader,
UnstructuredMarkdownLoader,
CSVLoader
)
# Load all markdown files
loader = DirectoryLoader(
"./knowledge",
glob="**/*.md",
loader_cls=UnstructuredMarkdownLoader
)
docs = loader.load()
# Load PDFs
pdf_loader = PyPDFLoader("document.pdf")
pdf_docs = pdf_loader.load()
# Load with metadata
for doc in docs:
doc.metadata["source_type"] = "knowledge_base"
doc.metadata["ingested_at"] = datetime.now().isoformat()
def enrich_metadata(doc):
"""Add useful metadata for filtering."""
content = doc.page_content
# Extract title from first heading
if content.startswith("#"):
doc.metadata["title"] = content.split("\n")[0].strip("# ")
# Estimate topic
doc.metadata["word_count"] = len(content.split())
# Add custom tags based on path
if "policy" in doc.metadata.get("source", ""):
doc.metadata["category"] = "policy"
return doc
docs = [enrich_metadata(doc) for doc in docs]
2. Chunking Strategies
Basic Chunking
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
length_function=len,
separators=["\n\n", "\n", " ", ""]
)
chunks = splitter.split_documents(docs)
Semantic Chunking
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
# Split by semantic similarity
semantic_splitter = SemanticChunker(
embeddings,
breakpoint_threshold_type="percentile",
breakpoint_threshold_amount=95
)
chunks = semantic_splitter.split_documents(docs)
Document-Aware Chunking
from langchain.text_splitter import MarkdownHeaderTextSplitter
# Split markdown by headers
headers_to_split_on = [
("#", "Header 1"),
("##", "Header 2"),
("###", "Header 3"),
]
markdown_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=headers_to_split_on
)
# Preserves header hierarchy in metadata
chunks = markdown_splitter.split_text(markdown_content)
Chunking Best Practices
| Factor |
Recommendation |
| Size |
500-1000 tokens for most use cases |
| Overlap |
10-20% to maintain context |
| Boundaries |
Respect semantic units |
| Metadata |
Preserve source, section info |
3. Embedding Selection
Model Comparison
| Model |
Dimensions |
Quality |
Speed |
Cost |
| text-embedding-3-large |
3072 |
Highest |
Medium |
$0.13/1M |
| text-embedding-3-small |
1536 |
Good |
Fast |
$0.02/1M |
| Cohere embed-v3 |
1024 |
High |
Fast |
$0.10/1M |
| BGE-large (local) |
1024 |
High |
Medium |
Free |
Implementation
from langchain_openai import OpenAIEmbeddings
from langchain_cohere import CohereEmbeddings
# OpenAI
embeddings = OpenAIEmbeddings(
model="text-embedding-3-small",
dimensions=1536 # Can reduce for cost savings
)
# Cohere
embeddings = CohereEmbeddings(
model="embed-english-v3.0",
input_type="search_document" # or "search_query"
)
# Local with HuggingFace
from langchain_huggingface import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(
model_name="BAAI/bge-large-en-v1.5"
)
4. Vector Storage
Store Comparison
| Store |
Best For |
Managed? |
Cost |
| Pinecone |
Production, simplicity |
Yes |
$70+/mo |
| Weaviate |
Hybrid search |
Both |
Open source |
| Qdrant |
Performance |
Both |
Open source |
| Chroma |
Development |
No |
Free |
| pgvector |
Postgres users |
No |
Free |
Pinecone Implementation
from langchain_pinecone import PineconeVectorStore
from pinecone import Pinecone
# Initialize
pc = Pinecone(api_key="your-key")
# Create index (once)
pc.create_index(
name="knowledge-base",
dimension=1536,
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1")
)
# Create vector store
vectorstore = PineconeVectorStore.from_documents(
chunks,
embeddings,
index_name="knowledge-base",
namespace="policies"
)
Chroma (Local Development)
from langchain_chroma import Chroma
# In-memory (dev)
vectorstore = Chroma.from_documents(chunks, embeddings)
# Persistent (prod)
vectorstore = Chroma.from_documents(
chunks,
embeddings,
persist_directory="./chroma_db"
)
5. Retrieval Strategies
Basic Retrieval
# Simple similarity search
retriever = vectorstore.as_retriever(
search_type="similarity",
search_kwargs={"k": 5}
)
docs = retriever.invoke("What is our refund policy?")
MMR (Diversity)
# Maximum Marginal Relevance - reduces redundancy
retriever = vectorstore.as_retriever(
search_type="mmr",
search_kwargs={
"k": 5,
"fetch_k": 20, # Fetch more, then diversify
"lambda_mult": 0.7 # 0=max diversity, 1=max relevance
}
)
Filtered Search
# Filter by metadata
retriever = vectorstore.as_retriever(
search_kwargs={
"k": 5,
"filter": {"category": "policy"}
}
)
Hybrid Search (BM25 + Vector)
from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
# BM25 for keyword matching
bm25 = BM25Retriever.from_documents(chunks)
bm25.k = 10
# Vector for semantic
vector = vectorstore.as_retriever(search_kwargs={"k": 10})
# Combine with weights
ensemble = EnsembleRetriever(
retrievers=[bm25, vector],
weights=[0.4, 0.6]
)
Reranking
from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank
# Get more docs initially
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 20})
# Rerank to top 5
reranker = CohereRerank(model="rerank-english-v3.0", top_n=5)
compression_retriever = ContextualCompressionRetriever(
base_compressor=reranker,
base_retriever=base_retriever
)
6. Generation
Basic QA Chain
from langchain.chains import RetrievalQA
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff", # Put all docs in context
retriever=retriever,
return_source_documents=True
)
result = qa_chain.invoke("What is our refund policy?")
print(result["result"])
print(result["source_documents"])
Custom Prompt
from langchain.prompts import PromptTemplate
prompt = PromptTemplate.from_template("""
Answer the question based only on the following context.
If the answer is not in the context, say "I don't have information about that."
Context:
{context}
Question: {question}
Answer: """)
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
retriever=retriever,
chain_type_kwargs={"prompt": prompt}
)
With Citations
from langchain.chains import create_citation_fuzzy_match_chain
# Automatically adds citations to response
citation_chain = create_citation_fuzzy_match_chain(llm)
# Or manual citation formatting
citation_prompt = PromptTemplate.from_template("""
Answer based on the context. Cite sources using [1], [2], etc.
Context:
{context}
Sources:
{sources}
Question: {question}
Answer with citations:""")
7. Evaluation
Retrieval Metrics
from ragas import evaluate
from ragas.metrics import (
context_precision,
context_recall,
faithfulness,
answer_relevancy
)
# Prepare test dataset
test_questions = [
{"question": "What is the refund policy?", "ground_truth": "..."},
# ... more test cases
]
# Run evaluation
results = evaluate(
dataset,
metrics=[
context_precision,
context_recall,
faithfulness,
answer_relevancy
]
)
print(results)
Manual Evaluation Checklist
## RAG Quality Review
For each test query:
### Retrieval Quality
- [ ] Relevant chunks retrieved?
- [ ] Important info not missed?
- [ ] Irrelevant chunks filtered?
### Generation Quality
- [ ] Answer is accurate?
- [ ] Answer is complete?
- [ ] No hallucinated details?
- [ ] Citations are correct?
### Edge Cases
- [ ] Handles "I don't know" appropriately?
- [ ] Handles ambiguous queries?
- [ ] Handles multi-part questions?
8. Production Considerations
Index Updates
def update_knowledge_base(new_docs):
"""Incremental update without full reindex."""
# Process new documents
chunks = splitter.split_documents(new_docs)
# Add to existing index
vectorstore.add_documents(chunks)
# Optional: Remove outdated docs
vectorstore.delete(
filter={"outdated": True}
)
Caching
from langchain.cache import RedisCache
import langchain
# Cache LLM responses
langchain.llm_cache = RedisCache(redis_url="redis://localhost:6379")
# Cache embeddings
from langchain.embeddings import CacheBackedEmbeddings
cached_embeddings = CacheBackedEmbeddings.from_bytes_store(
embeddings,
redis_store,
namespace="embeddings"
)
Monitoring
# Log retrieval quality
def log_retrieval(query, docs, response):
logger.info({
"query": query,
"num_docs": len(docs),
"top_doc_score": docs[0].metadata.get("score"),
"response_length": len(response)
})
Common Issues & Fixes
| Issue |
Cause |
Solution |
| Irrelevant results |
Poor chunking |
Improve chunk boundaries |
| Missing info |
Low k value |
Increase k, add reranking |
| Hallucinations |
Weak prompt |
Add "only use context" instruction |
| Slow queries |
Large index |
Add metadata filtering |
| Inconsistent quality |
Embedding mismatch |
Use same model for docs & queries |