Building a RAG System with Python: Step by Step

RAG combines information retrieval with AI generation, allowing a model to search relevant documents and use that information when generating responses. In this guide, you'll learn how RAG works and build your own application with Python.

What Is RAG?

RAG stands for Retrieval-Augmented Generation. Instead of asking an AI model to answer directly, a RAG system first searches a knowledge base for relevant information.

Example:

Question:"What is our company's refund policy?"

RAG bridges the gap between static AI knowledge and real-time, document-aware intelligence — turning any document collection into a searchable, answerable resource.

Core Components of a RAG System

Step 1: Prepare Documents

Every RAG system begins with data. The quality of your knowledge base directly determines the quality of the answers your system will generate.

Examples of document sources include:

Before indexing, clean your data: remove headers, footers, page numbers, and irrelevant formatting. Plain text works best. If you have HTML or Markdown, strip the markup. The cleaner your input, the more accurate your retrieval will be.

Step 2: Split Documents into Chunks

Large documents must be divided into smaller pieces for efficient retrieval. A chunk is a self-contained segment of text that can be independently retrieved and provided as context to the LLM.

1def chunk_text(text, chunk_size=200, overlap=20):
2    chunks = []
3    start = 0
4    while start < len(text):
5        end = min(start + chunk_size, len(text))
6        chunks.append(text[start:end])
7        start += chunk_size - overlap
8    return chunks

Step 3: Generate Embeddings

An embedding is a numerical vector representation of text. Words or sentences with similar meanings produce vectors that are close together in high-dimensional space. OpenAI's text-embedding-3-small converts any text into a 1536-dimensional vector that captures its semantic meaning.

1from openai import OpenAI
2
3client = OpenAI()
4
5response = client.embeddings.create(
6    model="text-embedding-3-small",
7    input="Python is a programming language."
8)
9
10embedding = response.data[0].embedding
11print(f"Vector dimension: {len(embedding)}")

Each chunk of your document gets its own embedding. These vectors are what the system searches through — finding the chunks whose embeddings are mathematically closest to the user's query embedding.

Step 4: Store Embeddings in a Vector Database

Popular options: Chroma, FAISS, Pinecone, Weaviate, Qdrant

A vector database stores embeddings and enables fast similarity search. When a query comes in, it compares the query embedding against all stored embeddings and returns the most similar ones — typically using cosine similarity.

1pip install chromadb

1import chromadb
2
3client = chromadb.Client()
4collection = client.create_collection(name="knowledge_base")
5
6# Add documents
7collection.add(
8    documents=[
9        "Python is a programming language.",
10        "Machine learning uses data."
11    ],
12    ids=["1", "2"]
13)
14
15# Search
16results = collection.query(
17    query_texts=["How is Python used?"],
18    n_results=2
19)
20print(results["documents"])

Step 5: Retrieve Relevant Documents

When a user asks a question, the system performs a semantic search:

1. Convert the user's question into an embedding
2. Search the vector database for the most similar chunks using cosine similarity
3. Retrieve the top K results (typically 3-10 chunks)

1def retrieve(query, collection, n=3):
2    # Embed the query
3    q_emb = client.embeddings.create(
4        model="text-embedding-3-small",
5        input=query
6    ).data[0].embedding
7    
8    # Search vector DB
9    results = collection.query(
10        query_embeddings=[q_emb],
11        n_results=n
12    )
13    return results["documents"][0]

The retrieved chunks become the context that grounds the LLM's response. Without this step, the model would rely solely on its training data — which may be outdated, incomplete, or unaware of your specific information.

Step 6: Send Context to the LLM

The final step combines everything: the retrieved chunks are injected into a prompt as context, and the LLM generates an answer grounded in that specific information. This is what separates RAG from a plain chatbot — the model answers based on your documents, not just its training data.

1from openai import OpenAI
2
3client = OpenAI()
4
5prompt = f"""
6Context:
7{context}
8
9Question:
10{question}
11
12Answer using only the provided context.
13"""
14
15response = client.responses.create(
16    model="gpt-4o",
17    input=prompt
18)
19
20print(response.output_text)

The prompt structure is critical. Notice the explicit instruction: "Answer using only the provided context." This prevents the model from hallucinating or pulling from its training data. If the context doesn't contain the answer, the model should say so rather than fabricating one.

Full RAG Pipeline

Example Project Structure

rag-project/
├── data/
│   ├── docs/
│   │   ├── guide.pdf
│   │   └── policies.txt
├── embeddings/
│   └── build_embeddings.py
├── vectorstore/
│   └── chroma_db/
├── rag/
│   ├── retrieve.py
│   ├── generate.py
│   └── pipeline.py
├── app.py
└── requirements.txt

Improving Retrieval Quality

Common Challenges

Real-World RAG Use Cases

Key Takeaways

Ready to go deeper?

Generative AI with Python

Master RAG pipelines, AI agents, tool calling, vector databases, and multimodal systems — with hands-on code throughout.

RAG & Vector DBsAI AgentsTool CallingMultimodal AI

Get it on Amazon →

Hernando Abella

Software engineer and author. I write about Python, AI, and software architecture. Author of 55+ programming books and creator of interactive coding challenges.