How to Build a Chatbot With the OpenAI API: A Developer's Practical Guide

Building a chatbot with the OpenAI API is one of the most direct paths from a working idea to a deployed product. The API is well-documented, the Python and Node.js SDKs are mature, and the core pattern — maintain a conversation history, send it to the API, return the response — is something you can have working in under an hour.

What takes longer is building a chatbot that is actually useful in a production context: one that stays on topic, handles edge cases gracefully, maintains context across long conversations, integrates with your business data, and does not cost a fortune to run at scale.

This guide covers the full picture: from the basic API call to a production-ready chatbot with system prompts, conversation memory, tool calls, streaming, and cost control. For the business use case this often serves — capturing and qualifying leads — see AI Chatbot for Lead Generation.

The Core Pattern

The OpenAI Chat Completions API takes an array of messages and returns the next message in the conversation. Each message has a role (system, user, or assistant) and content (the text of the message).

The simplest working chatbot in Python:

from openai import OpenAI

client = OpenAI(api_key="your-api-key")

conversation = [
    {"role": "system", "content": "You are a helpful assistant for Bilal's automation agency. Answer questions about Make.com, n8n, GoHighLevel, and AI voice agents."},
]

def chat(user_message: str) -> str:
    conversation.append({"role": "user", "content": user_message})
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=conversation,
    )
    
    assistant_message = response.choices[0].message.content
    conversation.append({"role": "assistant", "content": assistant_message})
    
    return assistant_message

# Usage
print(chat("What is Make.com good for?"))
print(chat("How does it compare to Zapier?"))

The conversation list grows with each exchange. The API always receives the full history, which is how the model maintains context. The user asks a follow-up question about "it" and the model knows what "it" refers to because it has seen the whole conversation.

The System Prompt: Where Chatbot Quality Is Made

The system message is the instruction set that shapes everything the chatbot does. It is the most important thing you configure and the part most developers underinvest in. The same discipline applies whether you're building a text chatbot or a voice agent — see How to Write a System Prompt for AI Agents for the deeper treatment of this skill.

A weak system prompt: "You are a helpful assistant."

A strong system prompt for a business chatbot:

You are an AI assistant for Riverside Dental, a family dental practice in Austin, Texas.

Your job is to help website visitors with:
- Questions about our services (cleanings, fillings, cosmetic dentistry, orthodontics)
- Appointment booking (collect name, phone, preferred date/time, service needed)
- Insurance questions (we accept Delta Dental, Cigna, Aetna, and United Healthcare)
- General dental FAQs

Our hours are Monday-Friday 8am-6pm and Saturday 9am-2pm.
Our address is 1234 Main Street, Austin TX 78701.
Our phone number is (512) 555-0100.

If someone asks about a specific tooth pain or dental emergency, provide basic first aid guidance and strongly encourage them to call us immediately or visit an emergency dentist.

Do not provide diagnoses. Do not discuss competitor practices. Do not discuss pricing without noting that costs vary by insurance and treatment plan.

Keep responses concise. If the visitor wants to book an appointment, collect their information and confirm you will pass it to our team.

This system prompt defines scope, persona, knowledge boundaries, and behaviour for specific situations. The chatbot built on it behaves consistently and appropriately because the instructions are specific.

Managing Conversation History for Long Chats

The OpenAI API has a context window limit. For GPT-4o mini it is 128,000 tokens. For most chatbot conversations this is more than sufficient, but long sessions or sessions with large system prompts can approach the limit.

The naive solution — just keep appending to the conversation list — eventually hits the limit and fails. A production chatbot needs conversation management.

Sliding window. Keep only the system prompt and the last N exchanges. Simple to implement, loses older context.

MAX_HISTORY = 20  # Keep last 10 exchanges (20 messages)

def trim_conversation(conversation: list) -> list:
    system_messages = [m for m in conversation if m["role"] == "system"]
    non_system = [m for m in conversation if m["role"] != "system"]
    return system_messages + non_system[-MAX_HISTORY:]

Summarisation. When the conversation exceeds a threshold, send the older portion to the API with a "summarise this conversation so far" prompt and replace the old messages with the summary. Maintains context without hitting limits.

Token counting. Use the tiktoken library to count tokens before each API call and trim or summarise based on actual token count rather than message count. More precise than message-based trimming.

import tiktoken

def count_tokens(messages: list, model: str = "gpt-4o-mini") -> int:
    encoding = tiktoken.encoding_for_model(model)
    return sum(len(encoding.encode(m["content"])) for m in messages)

Streaming Responses

For any chatbot where the user sees the response as it is generated — which is the standard experience in web and chat interfaces — streaming is essential. Without streaming, the user stares at a blank screen until the full response is ready. With streaming, they see tokens appearing as the model generates them.

def chat_streaming(user_message: str):
    conversation.append({"role": "user", "content": user_message})
    
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=conversation,
        stream=True,
    )
    
    full_response = ""
    for chunk in stream:
        if chunk.choices[0].delta.content:
            token = chunk.choices[0].delta.content
            full_response += token
            print(token, end="", flush=True)  # Or send to your frontend
    
    conversation.append({"role": "assistant", "content": full_response})
    return full_response

For web applications, server-sent events (SSE) from a FastAPI endpoint deliver the stream to the browser in real time. The frontend displays each token as it arrives. Python FastAPI Webhook Automation covers the broader FastAPI patterns this deployment builds on.

Tool Calls: Giving the Chatbot Actions

A chatbot that can only converse is limited. A chatbot that can take actions — look up data, create records, send messages, check availability — is genuinely useful for business automation.

The OpenAI tool calling (function calling) system lets you define functions that the model can choose to invoke, and OpenAI's function calling guide documents the schema those definitions have to follow. When the model decides a tool call is appropriate, it returns a structured tool call object instead of a text response. Your code executes the function and passes the result back to the model.

tools = [
    {
        "type": "function",
        "function": {
            "name": "check_appointment_availability",
            "description": "Check available appointment slots for a given date",
            "parameters": {
                "type": "object",
                "properties": {
                    "date": {
                        "type": "string",
                        "description": "The date to check availability for, in YYYY-MM-DD format"
                    },
                    "service_type": {
                        "type": "string",
                        "description": "The type of appointment requested"
                    }
                },
                "required": ["date"]
            }
        }
    }
]

def handle_tool_call(tool_name: str, tool_args: dict) -> str:
    if tool_name == "check_appointment_availability":
        # Your actual availability check logic here
        date = tool_args.get("date")
        return f"Available slots on {date}: 9:00am, 11:00am, 2:00pm, 4:00pm"
    return "Tool not found"

def chat_with_tools(user_message: str) -> str:
    conversation.append({"role": "user", "content": user_message})
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=conversation,
        tools=tools,
    )
    
    message = response.choices[0].message
    
    # Check if the model wants to call a tool
    if message.tool_calls:
        conversation.append(message)  # Add the tool call message
        
        for tool_call in message.tool_calls:
            import json
            result = handle_tool_call(
                tool_call.function.name,
                json.loads(tool_call.function.arguments)
            )
            conversation.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": result
            })
        
        # Get the final response with tool results included
        final_response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=conversation,
        )
        assistant_content = final_response.choices[0].message.content
        conversation.append({"role": "assistant", "content": assistant_content})
        return assistant_content
    
    conversation.append({"role": "assistant", "content": message.content})
    return message.content

Cost Control

At scale, OpenAI API costs add up. Three practices that keep costs manageable.

Use the right model. Model names and per-token prices move often enough that any figure written down here will be wrong before long, so check OpenAI's API pricing page before you size anything. The shape of it has been stable though: the small model in each generation costs an order of magnitude less than the large one and handles the majority of chatbot use cases well. Use GPT-4o for complex reasoning tasks. Use GPT-4o mini for everything else. The quality difference for typical customer-facing chatbot conversations is minimal.

Cache frequent responses. If your chatbot answers the same questions repeatedly — business hours, pricing, return policy — cache the responses in Redis or a simple dictionary. Return the cached response for matching inputs without an API call.

Monitor usage. The OpenAI dashboard shows usage by day, by model, and by organisation. Set usage limits and alerts. A runaway automation scenario or an unexpected traffic spike should not result in a surprise bill.

The OpenAI Cost Calculator models your expected monthly API cost at different usage volumes and model combinations before you commit to an architecture.

Deploying as a FastAPI Endpoint

Wrapping the chatbot in a FastAPI endpoint makes it accessible to any frontend or integration.

from fastapi import FastAPI
from pydantic import BaseModel
from fastapi.responses import StreamingResponse
import json

app = FastAPI()

class ChatRequest(BaseModel):
    message: str
    session_id: str

sessions = {}  # In production, use Redis

@app.post("/chat")
async def chat_endpoint(request: ChatRequest):
    if request.session_id not in sessions:
        sessions[request.session_id] = [
            {"role": "system", "content": "Your system prompt here"}
        ]
    
    conversation = sessions[request.session_id]
    conversation.append({"role": "user", "content": request.message})
    
    async def generate():
        stream = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=conversation,
            stream=True,
        )
        full_response = ""
        for chunk in stream:
            if chunk.choices[0].delta.content:
                token = chunk.choices[0].delta.content
                full_response += token
                yield f"data: {json.dumps({'token': token})}\n\n"
        conversation.append({"role": "assistant", "content": full_response})
        yield "data: [DONE]\n\n"
    
    return StreamingResponse(generate(), media_type="text/event-stream")

This endpoint maintains session state per session ID, streams the response, and updates the conversation history. In production, replace the in-memory sessions dictionary with Redis for persistence across server restarts and multi-instance deployments.

What to Build Next

Once the core chatbot is working, the most valuable additions are: a knowledge base using retrieval-augmented generation (RAG) so the chatbot can answer questions from your specific documents — see How to Build a RAG System for that build — guardrails that detect and handle off-topic or inappropriate requests, and analytics that track which questions are asked most frequently so you can improve the system prompt over time.

If you want help designing a production chatbot for a specific business use case or need a review of a chatbot architecture you are building, book a free 30-minute call. Bring the use case and the current architecture and we will work through the design together.

Frequently Asked Questions

Why are two different users seeing each other's conversation?

Because the history is a module-level variable. The short examples above keep conversation at module scope to stay readable, and that is fine at a prompt in your terminal and catastrophic the moment two people hit the same process. Every request appends to the same list, so user B receives the model's answer shaped by user A's messages, and in a support context that is a data leak rather than a bug. Key the history by session from the first line of real code, as the FastAPI example does, and never let a request handler read conversation state that was not passed into it.

Why does the cost climb the longer a conversation runs?

Because you resend the entire history on every turn, so the input tokens for turn ten include turns one through nine. Cost per exchange grows with conversation length even though each user message stays the same size, and a long session ends up costing far more than the same number of messages spread across separate sessions. Trimming or summarising fixes most of it. Where the providers offer caching on a repeated prefix, keeping a long system prompt stable rather than rebuilding it per request lets that discount apply.

Can a user override my system prompt?

Often, yes. The system message shapes behaviour but it is not a security boundary, and instructions arriving in the user turn compete with it rather than being subordinate to it. Anyone determined enough will eventually get the model to ignore parts of the prompt. Design so that it does not matter: enforce limits in your own code rather than in prose, keep secrets and internal data out of the prompt entirely, and validate tool call arguments server-side before executing anything. "Only look up bookings belonging to this user" belongs in the query, not in the instructions.

What should happen when the API times out mid-stream?

Decide before it happens, because it will. A stream that dies halfway leaves the user with a truncated sentence and leaves your history holding a partial assistant message that will confuse the next turn. Set an explicit timeout, catch the failure, and either discard the partial response or store it clearly marked as incomplete. Retry once on transient errors and rate limits with a short backoff, but do not retry blind on a request that already triggered a tool call, or you will run the side effect twice.

Is an in-memory session dictionary really a problem?

Yes, in two ways. It never releases anything, so a busy service grows until the process is killed, and it belongs to one instance, so the moment you run two the same user gets a different memory depending on which one answers. Move sessions to Redis with an expiry that matches how long a conversation plausibly stays alive, an hour or so for most support use cases. Storing them there also means a deploy no longer wipes every conversation in progress.

If you would rather have this built than build it, I take on chatbot and API integration work through Fiverr.