Retell AI Custom LLM Integration: How to Connect Your Own Model to a Voice Agent

Retell AI's built-in agent builder covers a wide range of voice agent use cases. But when your use case requires a specific model, a fine-tuned LLM, a custom inference endpoint, or logic that the built-in builder cannot express cleanly, the Custom LLM integration is the path forward. It lets you bring your own language model to the Retell voice stack while Retell continues to handle everything else: telephony, speech-to-text, text-to-speech, latency optimisation, and the call management layer.

This post covers how the Custom LLM integration works, how to build the server Retell expects, and how to handle the parts that require careful design. If you're new to Retell entirely, How to Use Retell AI is the place to start first.

What the Custom LLM Integration Actually Does

In a standard Retell agent, the LLM layer is managed by Retell. You provide a system prompt and Retell sends each conversation turn to an OpenAI or Anthropic model on your behalf.

With Custom LLM, you provide an HTTP endpoint that Retell calls instead. Retell sends the conversation history to your endpoint in a defined format. Your endpoint processes it — using any model, any logic, any external data lookups — and returns a response in the format Retell expects. Retell takes that response and converts it to speech for the caller.

From the caller's perspective, nothing changes. From the developer's perspective, you now have full control over the LLM layer: which model is used, what context is injected, how tool calls are handled, what happens when specific conditions are met.

When to Use Custom LLM vs the Built-In Builder

The built-in Retell LLM is the right choice for most standard voice agent builds. It is faster to set up, requires no backend infrastructure, and handles the common patterns — qualification, booking, FAQ handling — well.

Reach for Custom LLM when:

You need a specific model. The built-in builder uses OpenAI and Anthropic models via Retell's own API keys. If you need a fine-tuned model, a specific model version with guaranteed availability, a locally hosted open-source model, or a model from a provider not available in the built-in builder, Custom LLM is the only option.

You need real-time data lookups mid-conversation. Tool calls in the built-in builder can trigger external requests, but the logic around when to call tools and how to handle responses is constrained. A custom LLM endpoint can run arbitrary code between model calls — looking up pricing in a database, checking a patient record, querying a property listing feed — with full control over when and how that happens. This is the same real-time listing lookup pattern covered in AI Voice Agent for Real Estate.

Your conversation logic is complex enough that prompt engineering alone is not sufficient. Some call flows have enough conditional branches, state requirements, or dynamic content that managing them entirely through a system prompt becomes fragile. A custom backend lets you manage state programmatically and inject exactly the right context at each turn.

You need to log, modify, or monitor every conversation turn. A custom endpoint sits in the middle of every exchange, giving you a place to log, validate, modify, or audit the full conversation in real time.

The Request Format Retell Sends

When Retell calls your custom LLM endpoint, it sends a POST request with a JSON body containing the full conversation history in a format similar to the OpenAI Chat Completions API.

The body looks like this:

{
  "model": "your-custom-model",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant for Riverside Clinic..."
    },
    {
      "role": "user",
      "content": "Hi, I need to book an appointment"
    },
    {
      "role": "assistant",
      "content": "Of course. Are you an existing patient or a new patient?"
    },
    {
      "role": "user",
      "content": "New patient"
    }
  ],
  "stream": true,
  "call": {
    "call_id": "abc123",
    "agent_id": "agent_xyz",
    "metadata": {}
  }
}

The messages array contains the full conversation history from the current call, including system prompt, all prior assistant turns, and all caller utterances. Your endpoint has the complete context every time it is called.

The call object contains the call ID and any metadata you passed when initiating the call. This is how you pass call-specific data — a patient ID, a lead source, a property address — into your endpoint for use in the LLM call or data lookups.

The Response Format Retell Expects

Retell expects a streaming response in Server-Sent Events (SSE) format, structured like the OpenAI streaming chat completions response. Each chunk contains a delta with the next token or tokens of the response.

For a non-streaming response (simpler but higher latency), Retell also accepts a standard JSON response with the full assistant message. For production voice agents where latency matters, streaming is strongly preferred — it allows Retell to begin converting text to speech before your full response is complete, which meaningfully reduces the perceived response time.

If your response includes a tool call, format it in the OpenAI tool call format within the streaming response. Retell will detect the tool call, execute it via your configured tool definitions, and pass the tool result back to your endpoint in the next request as a tool message in the messages array.

Building the Custom LLM Endpoint in FastAPI

Here is a minimal working implementation in FastAPI that proxies to OpenAI while giving you a place to add custom logic. Python FastAPI Webhook Automation covers the broader backend patterns this builds on.

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import openai
import json
import os

app = FastAPI()
client = openai.AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])

@app.post("/llm")
async def custom_llm(request: Request):
    body = await request.json()
    messages = body.get("messages", [])
    call_metadata = body.get("call", {}).get("metadata", {})

    # Add dynamic context based on call metadata
    # e.g. inject patient info, property details, lead data
    if call_metadata.get("patient_id"):
        patient_context = await fetch_patient_context(call_metadata["patient_id"])
        messages[0]["content"] += f"\n\nPatient context: {patient_context}"

    async def generate():
        stream = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            stream=True,
        )
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                data = {
                    "choices": [{
                        "delta": {"content": chunk.choices[0].delta.content},
                        "finish_reason": chunk.choices[0].finish_reason
                    }]
                }
                yield f"data: {json.dumps(data)}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        generate(),
        media_type="text/event-stream"
    )

async def fetch_patient_context(patient_id: str) -> str:
    # Your database or API lookup here
    return f"Patient ID {patient_id}: returning patient, last visit 3 months ago"

This is the scaffold. The real logic goes into the context injection, the system prompt construction, and any tool call handling. The streaming format handles the Retell response requirements.

Injecting Dynamic Context Per Call

The metadata field in the Retell call object is the mechanism for passing call-specific data into your custom LLM endpoint. When you initiate an outbound call or configure an inbound agent, you pass a metadata object with whatever key-value pairs are relevant.

For a dental clinic, you might pass the patient ID when an appointment reminder triggers an outbound call — see Voice AI for Dental Clinics for the broader context. Your endpoint retrieves the patient record and injects it into the system prompt before the first LLM call. The agent now knows the patient name, their last appointment, their provider, and their upcoming appointment details — without that information being hardcoded in the agent configuration.

For a real estate voice agent, you pass the listing ID when a caller inquires about a specific property. Your endpoint fetches the live listing data and injects it into the context. The agent answers questions about the specific property with current, accurate information.

This pattern — call initiation passes an ID, the custom LLM endpoint fetches context from a database or API, context is injected per turn — is what separates a generic voice agent from one that feels like it genuinely knows the caller and their situation.

Latency Considerations

Latency is the most important metric for voice agents. Every millisecond of processing time in your custom LLM endpoint adds to the perceived response delay the caller experiences.

Optimise for latency at every step:

Use the fastest model that meets your quality requirement. GPT-4o mini is significantly faster than GPT-4o. For most voice agent turns, the quality difference is minimal. Use the faster model as the default and only escalate to a more capable model for specific complex turns if needed.

Cache external data lookups. If your endpoint fetches patient records or property data on every turn, cache the results in memory after the first fetch. The data does not change mid-call. Hitting the database on turn one and reusing the cached result on turns two through ten cuts lookup latency for 90 percent of turns.

Start streaming immediately. Do not buffer the full response before streaming. Return the first token as soon as the model produces it. This allows Retell to begin TTS conversion while your model is still generating, which reduces the end-to-end latency the caller perceives.

Deploy close to Retell's infrastructure. Retell runs on US-based infrastructure. Deploying your custom LLM endpoint in a US data centre (or the closest available region) minimises network round-trip time.

Testing Your Custom LLM Endpoint

Before connecting to Retell, test the endpoint directly. Use the API Request Tester or curl to send a sample Retell-format request body to your endpoint and confirm the streaming response comes back in the correct SSE format with the right structure.

Once the endpoint is responding correctly, add it to Retell in the agent configuration under Custom LLM. Run test calls through the Retell dashboard's built-in web call feature. Listen to the call quality and watch the call transcripts. The first few calls almost always reveal places where the context injection or the system prompt needs adjustment.

What This Unlocks

The Retell AI Custom LLM integration is where voice agent development stops being configuration and starts being engineering. You have a voice layer that handles the hardest parts of real-time phone AI, and a fully programmable LLM layer where you write the logic.

The combination is genuinely powerful: a production-grade voice stack that most teams would take months to build from scratch, paired with a custom backend that gives you full control over what the agent knows, what it can do, and how it reasons about each call. This is the same architecture pattern I've used to build dynamic listing lookups and patient-context injection for voice agent clients in real estate and healthcare.

If you want help designing the custom LLM architecture for a specific voice agent build or need a working implementation reviewed, book a free 30-minute call. Bring the use case and we will work through the integration design together.