How to Build a RAG System: Retrieval-Augmented Generation for Business AI Applications

Retrieval-augmented generation (RAG) is the architecture that makes AI genuinely useful for business-specific knowledge. Without it, an AI system can only answer questions based on what it learned during training — general knowledge, but nothing specific to your company, your products, your policies, or your documents. With RAG, the AI retrieves relevant information from your own knowledge base before generating a response, grounding its answers in your actual content.

This post covers how RAG works, when to use it, and how to build a practical RAG system for a business application. It's the natural next step once you have a working chatbot — see How to Build a Chatbot With the OpenAI API for the base architecture this extends.

What RAG Solves

The core limitation of a standard LLM for business use is that it does not know anything specific to your business. Ask GPT-4o about a general topic and it answers well. Ask it about your company's refund policy, your specific product specifications, or a client agreement you signed last month and it either makes something up or tells you it does not know.

RAG solves this by adding a retrieval step before the generation step. When a user asks a question, the system first searches your knowledge base for the most relevant documents or passages. Those passages are injected into the LLM prompt as context. The LLM then generates a response grounded in that specific content rather than relying on general training knowledge.

The result: an AI that answers questions about your business accurately, using your actual documents as the source of truth. This is the same architecture behind the support deflection layer covered in How to Reduce Customer Support Costs With AI.

The Three Components of a RAG System

1. The Knowledge Base

The knowledge base is the collection of documents the system retrieves from. This can be anything text-based: product documentation, FAQs, policy documents, support articles, training materials, client contracts, meeting notes, SOPs.

The quality of the knowledge base directly determines the quality of the RAG system. Well-written, clearly structured documents produce better retrieval results than messy, poorly formatted ones. Deduplicate content and remove outdated documents before ingestion.

2. The Vector Store

A vector store is a specialised database that stores documents as numerical vectors (embeddings) and enables semantic search. Unlike keyword search, which matches exact terms, semantic search finds documents that are conceptually similar to the query even when they use different words.

When documents are ingested, an embedding model converts each document chunk into a vector — a list of numbers that represents its meaning. OpenAI's embeddings guide documents the models that produce those vectors and how to call them. These vectors are stored in the vector database.

When a query arrives, the same embedding model converts the query into a vector. The vector database finds the stored document vectors most similar to the query vector and returns those documents as search results.

Popular vector stores: Pinecone (managed, cloud-based), Weaviate (open-source, self-hostable), Qdrant (open-source, fast), Chroma (lightweight, good for development), and pgvector (PostgreSQL extension for teams already using Postgres).

3. The Generation Layer

The generation layer is the LLM that produces the final response. It receives the user query and the retrieved document passages as input and generates a response that synthesises the relevant information.

The prompt structure for RAG generation looks like this:

You are a helpful assistant for [Company]. Answer the user's question based only on the provided context. If the context does not contain enough information to answer the question, say so clearly.

Context:
[Retrieved document passages go here]

User question: [The user's actual question]

The instruction to answer only from the provided context is critical. Without it, the LLM will blend retrieved information with its general training knowledge, which can introduce inaccuracies for business-specific queries. How to Write a System Prompt for AI Agents covers this kind of constraint-writing in more depth.

Building a Basic RAG System in Python

Here is a practical implementation using OpenAI for embeddings and generation, and Chroma as a lightweight local vector store. The Chroma documentation covers the client and collection API used below.

Step 1: Install dependencies

pip install openai chromadb tiktoken

Step 2: Ingest documents into the vector store

import chromadb
from openai import OpenAI
import os

openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
chroma_client = chromadb.Client()
collection = chroma_client.create_collection("knowledge_base")

def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Split text into overlapping chunks."""
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

def embed_text(text: str) -> list[float]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def ingest_document(document_text: str, document_id: str, metadata: dict = None):
    chunks = chunk_text(document_text)
    for i, chunk in enumerate(chunks):
        embedding = embed_text(chunk)
        collection.add(
            embeddings=[embedding],
            documents=[chunk],
            ids=[f"{document_id}_chunk_{i}"],
            metadatas=[metadata or {}]
        )

# Ingest your documents
ingest_document(
    "Our refund policy allows returns within 30 days of purchase. Items must be in original condition with receipt.",
    "refund-policy",
    {"source": "policy", "category": "returns"}
)

Step 3: Retrieve and generate

def retrieve_relevant_chunks(query: str, n_results: int = 3) -> list[str]:
    query_embedding = embed_text(query)
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results
    )
    return results["documents"][0]

def answer_question(question: str) -> str:
    relevant_chunks = retrieve_relevant_chunks(question)
    context = "\n\n".join(relevant_chunks)
    
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": f"""You are a helpful assistant. Answer the user's question based only on the provided context. If the context does not contain enough information, say so.

Context:
{context}"""
            },
            {
                "role": "user",
                "content": question
            }
        ]
    )
    return response.choices[0].message.content

# Usage
print(answer_question("What is your return policy?"))

Chunking Strategy Matters

How you split documents into chunks significantly affects retrieval quality. Poor chunking produces poor retrieval.

Fixed-size chunking (as shown above) is simple but can split semantically coherent passages across chunk boundaries. A sentence that provides critical context might be split across two chunks, neither of which retrieves accurately.

Semantic chunking splits on natural boundaries: paragraphs, sections, or sentences. This preserves semantic coherence at the cost of variable chunk sizes.

Hierarchical chunking stores both small chunks (for precise retrieval) and larger parent chunks (for complete context). The small chunk is retrieved based on similarity; the large parent chunk is sent to the LLM for context. This is one of the more effective approaches for complex documents.

For most business RAG applications, paragraph-level chunking with 200 to 500 words per chunk and 10 to 20 percent overlap between chunks is a solid starting point.

Improving Retrieval Quality

Basic semantic search works well for clear questions against well-written documents. Retrieval quality degrades when queries are ambiguous, when documents are poorly structured, or when the relevant information is expressed in different terminology from the query.

Hybrid search. Combine vector search with keyword (BM25) search and merge the results. Hybrid search handles cases where exact term matching outperforms semantic similarity.

Query expansion. Before retrieving, use an LLM to generate alternative phrasings of the user's query. Retrieve for each phrasing and combine the results. This catches relevant documents that use different terminology.

Re-ranking. After initial retrieval, use a cross-encoder model to re-score the retrieved chunks for relevance to the query. Cross-encoders are slower than vector similarity but more accurate. Apply them only to the top N initial results.

Metadata filtering. If your documents have metadata (category, date, source), filter by metadata before semantic search. A query about "return policy" filtered to category:policy retrieves more precisely than an unfiltered search across all documents.

Production Considerations

Managed vector store. For production, use a managed vector store (Pinecone, Weaviate Cloud, Qdrant Cloud) rather than an in-memory local store. These persist data, scale horizontally, and provide reliable uptime. If you go that way, the Pinecone documentation is the place to start.

Embedding model selection. OpenAI's text-embedding-3-small is a good default: fast, inexpensive, and high quality. For multilingual knowledge bases, use a model with strong multilingual support.

Caching. Cache embeddings for documents that do not change. Re-embedding the entire knowledge base on every application restart is wasteful. Store embeddings in the vector store and only re-embed when documents change.

Evaluation. Test your RAG system against a set of known question-answer pairs. Measure retrieval recall (are the right documents being retrieved?) and answer accuracy (is the generated answer correct?). Iterate on chunking strategy, retrieval parameters, and prompt structure based on the evaluation results.

The Knowledge Base Generator helps structure and draft the content that feeds into a RAG system — useful for creating clean, well-organised source documents before ingestion.

When RAG Is the Right Architecture

RAG is the right approach when: the AI needs to answer questions about your specific documents or data, the knowledge base changes over time and needs to stay current, hallucination risk is high and groundedness in real sources is critical, or you need the system to cite sources for its answers.

RAG is not the right approach when: the knowledge is static and small enough to fit in a context window (just include it in the system prompt), the queries do not relate to specific documents, or the latency of a retrieval step is unacceptable for the use case.

If you want help designing and building a RAG system for a specific business application — customer support, internal knowledge base, document Q&A — book a free 30-minute call. Bring the documents you want to make queryable and the questions you want the system to answer, and we will design the right architecture together.

Frequently Asked Questions

Does RAG stop the model making things up?

It reduces it substantially. It does not eliminate it, and the instruction to answer only from context is guidance rather than a guarantee. The failure most people miss is that retrieval always returns something: ask about a topic your knowledge base has never covered and the vector search still hands back the three least-irrelevant chunks it has, at which point the model does its best with material that does not answer the question. Check the similarity score before you generate and refuse below a threshold. Asking for the source alongside the answer helps too, because a citation that does not support the claim is visible in a way a confident paragraph is not.

How do I update or delete a document once it is ingested?

Deliberately, because the naive path leaves you with both versions. Chunks are stored under generated ids, so re-ingesting an edited document writes a fresh set alongside the old one unless you remove the previous chunks first, and a document that shrank leaves orphaned chunks with no parent. Give every chunk a metadata field naming its source document, delete by that field before re-ingesting, and store a content hash so you can skip documents that have not actually changed. A retrieval system quietly serving last quarter's policy is worse than having no system.

Can the system retrieve a document the person asking is not allowed to see?

Yes, unless you have specifically prevented it, and this is the most common serious mistake in internal RAG deployments. Semantic search has no concept of who is asking; it returns whatever is closest to the query, salary reviews and board papers included. Attach permissions as metadata at ingest and filter on them inside the query rather than trimming the results afterwards, since anything retrieved has already reached the prompt. Where documents have genuinely different audiences, separate collections are easier to reason about than one collection with careful filters.

What happens if I switch embedding model?

You re-embed everything. Vectors produced by different models are not comparable, and mixing them in one collection does not error, it just returns nonsense rankings, which is far harder to notice than a crash. Plan the migration as a rebuild: create a new collection, ingest the full corpus with the new model, verify retrieval against your test questions, then switch over. Keep a note of which model built which collection, because working this out later from the vector dimensions alone is unpleasant.

Do I actually need a vector database?

For a few dozen documents, probably not. If the whole knowledge base fits comfortably inside a context window, putting it in the prompt is simpler, has no retrieval step to get wrong, and lets the model see everything rather than the three chunks that scored highest. Below a few hundred chunks, embeddings held in memory and compared directly are also perfectly adequate and remove an entire piece of infrastructure. The vector database earns its place when the corpus is too large for the context window, changes often, or needs metadata filtering and access control.

If you would rather have this built than build it, I take on RAG and AI integration work through Upwork.