guide

Implementation Guide: RAG Systems

Build production RAG systems. Covers chunking strategies, embedding selection, retrieval optimization, and evaluation methods.

Overview

RAG (Retrieval-Augmented Generation) enhances LLM responses by retrieving relevant context from a knowledge base. This guide covers production implementation.

RAG Pipeline

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Ingest   │──►│ Chunk    │──►│ Embed    │──►│ Store    │──►│ Index    β”‚
β”‚ Documentsβ”‚   β”‚          β”‚   β”‚          β”‚   β”‚ Vectors  β”‚   β”‚          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Query    │──►│ Embed    │──►│ Search   │──►│ Rerank   │──►│ Generate β”‚
β”‚          β”‚   β”‚ Query    β”‚   β”‚          β”‚   β”‚          β”‚   β”‚          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

1. Document Ingestion

Loading Documents

from langchain_community.document_loaders import (
    DirectoryLoader,
    PyPDFLoader,
    UnstructuredMarkdownLoader,
    CSVLoader
)

# Load all markdown files
loader = DirectoryLoader(
    "./knowledge",
    glob="**/*.md",
    loader_cls=UnstructuredMarkdownLoader
)
docs = loader.load()

# Load PDFs
pdf_loader = PyPDFLoader("document.pdf")
pdf_docs = pdf_loader.load()

# Load with metadata
for doc in docs:
    doc.metadata["source_type"] = "knowledge_base"
    doc.metadata["ingested_at"] = datetime.now().isoformat()

Metadata Enrichment

def enrich_metadata(doc):
    """Add useful metadata for filtering."""
    content = doc.page_content

    # Extract title from first heading
    if content.startswith("#"):
        doc.metadata["title"] = content.split("\n")[0].strip("# ")

    # Estimate topic
    doc.metadata["word_count"] = len(content.split())

    # Add custom tags based on path
    if "policy" in doc.metadata.get("source", ""):
        doc.metadata["category"] = "policy"

    return doc

docs = [enrich_metadata(doc) for doc in docs]

2. Chunking Strategies

Basic Chunking

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    separators=["\n\n", "\n", " ", ""]
)

chunks = splitter.split_documents(docs)

Semantic Chunking

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

# Split by semantic similarity
semantic_splitter = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95
)

chunks = semantic_splitter.split_documents(docs)

Document-Aware Chunking

from langchain.text_splitter import MarkdownHeaderTextSplitter

# Split markdown by headers
headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on
)

# Preserves header hierarchy in metadata
chunks = markdown_splitter.split_text(markdown_content)

Chunking Best Practices

Factor Recommendation
Size 500-1000 tokens for most use cases
Overlap 10-20% to maintain context
Boundaries Respect semantic units
Metadata Preserve source, section info

3. Embedding Selection

Model Comparison

Model Dimensions Quality Speed Cost
text-embedding-3-large 3072 Highest Medium $0.13/1M
text-embedding-3-small 1536 Good Fast $0.02/1M
Cohere embed-v3 1024 High Fast $0.10/1M
BGE-large (local) 1024 High Medium Free

Implementation

from langchain_openai import OpenAIEmbeddings
from langchain_cohere import CohereEmbeddings

# OpenAI
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    dimensions=1536  # Can reduce for cost savings
)

# Cohere
embeddings = CohereEmbeddings(
    model="embed-english-v3.0",
    input_type="search_document"  # or "search_query"
)

# Local with HuggingFace
from langchain_huggingface import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-large-en-v1.5"
)

4. Vector Storage

Store Comparison

Store Best For Managed? Cost
Pinecone Production, simplicity Yes $70+/mo
Weaviate Hybrid search Both Open source
Qdrant Performance Both Open source
Chroma Development No Free
pgvector Postgres users No Free

Pinecone Implementation

from langchain_pinecone import PineconeVectorStore
from pinecone import Pinecone

# Initialize
pc = Pinecone(api_key="your-key")

# Create index (once)
pc.create_index(
    name="knowledge-base",
    dimension=1536,
    metric="cosine",
    spec=ServerlessSpec(cloud="aws", region="us-east-1")
)

# Create vector store
vectorstore = PineconeVectorStore.from_documents(
    chunks,
    embeddings,
    index_name="knowledge-base",
    namespace="policies"
)

Chroma (Local Development)

from langchain_chroma import Chroma

# In-memory (dev)
vectorstore = Chroma.from_documents(chunks, embeddings)

# Persistent (prod)
vectorstore = Chroma.from_documents(
    chunks,
    embeddings,
    persist_directory="./chroma_db"
)

5. Retrieval Strategies

Basic Retrieval

# Simple similarity search
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 5}
)

docs = retriever.invoke("What is our refund policy?")

MMR (Diversity)

# Maximum Marginal Relevance - reduces redundancy
retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={
        "k": 5,
        "fetch_k": 20,  # Fetch more, then diversify
        "lambda_mult": 0.7  # 0=max diversity, 1=max relevance
    }
)
# Filter by metadata
retriever = vectorstore.as_retriever(
    search_kwargs={
        "k": 5,
        "filter": {"category": "policy"}
    }
)

Hybrid Search (BM25 + Vector)

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

# BM25 for keyword matching
bm25 = BM25Retriever.from_documents(chunks)
bm25.k = 10

# Vector for semantic
vector = vectorstore.as_retriever(search_kwargs={"k": 10})

# Combine with weights
ensemble = EnsembleRetriever(
    retrievers=[bm25, vector],
    weights=[0.4, 0.6]
)

Reranking

from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank

# Get more docs initially
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 20})

# Rerank to top 5
reranker = CohereRerank(model="rerank-english-v3.0", top_n=5)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=base_retriever
)

6. Generation

Basic QA Chain

from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",  # Put all docs in context
    retriever=retriever,
    return_source_documents=True
)

result = qa_chain.invoke("What is our refund policy?")
print(result["result"])
print(result["source_documents"])

Custom Prompt

from langchain.prompts import PromptTemplate

prompt = PromptTemplate.from_template("""
Answer the question based only on the following context.
If the answer is not in the context, say "I don't have information about that."

Context:
{context}

Question: {question}

Answer: """)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    chain_type_kwargs={"prompt": prompt}
)

With Citations

from langchain.chains import create_citation_fuzzy_match_chain

# Automatically adds citations to response
citation_chain = create_citation_fuzzy_match_chain(llm)

# Or manual citation formatting
citation_prompt = PromptTemplate.from_template("""
Answer based on the context. Cite sources using [1], [2], etc.

Context:
{context}

Sources:
{sources}

Question: {question}

Answer with citations:""")

7. Evaluation

Retrieval Metrics

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy
)

# Prepare test dataset
test_questions = [
    {"question": "What is the refund policy?", "ground_truth": "..."},
    # ... more test cases
]

# Run evaluation
results = evaluate(
    dataset,
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy
    ]
)

print(results)

Manual Evaluation Checklist

## RAG Quality Review

For each test query:

### Retrieval Quality
- [ ] Relevant chunks retrieved?
- [ ] Important info not missed?
- [ ] Irrelevant chunks filtered?

### Generation Quality
- [ ] Answer is accurate?
- [ ] Answer is complete?
- [ ] No hallucinated details?
- [ ] Citations are correct?

### Edge Cases
- [ ] Handles "I don't know" appropriately?
- [ ] Handles ambiguous queries?
- [ ] Handles multi-part questions?

8. Production Considerations

Index Updates

def update_knowledge_base(new_docs):
    """Incremental update without full reindex."""
    # Process new documents
    chunks = splitter.split_documents(new_docs)

    # Add to existing index
    vectorstore.add_documents(chunks)

    # Optional: Remove outdated docs
    vectorstore.delete(
        filter={"outdated": True}
    )

Caching

from langchain.cache import RedisCache
import langchain

# Cache LLM responses
langchain.llm_cache = RedisCache(redis_url="redis://localhost:6379")

# Cache embeddings
from langchain.embeddings import CacheBackedEmbeddings

cached_embeddings = CacheBackedEmbeddings.from_bytes_store(
    embeddings,
    redis_store,
    namespace="embeddings"
)

Monitoring

# Log retrieval quality
def log_retrieval(query, docs, response):
    logger.info({
        "query": query,
        "num_docs": len(docs),
        "top_doc_score": docs[0].metadata.get("score"),
        "response_length": len(response)
    })

Common Issues & Fixes

Issue Cause Solution
Irrelevant results Poor chunking Improve chunk boundaries
Missing info Low k value Increase k, add reranking
Hallucinations Weak prompt Add "only use context" instruction
Slow queries Large index Add metadata filtering
Inconsistent quality Embedding mismatch Use same model for docs & queries