GPT Realtime Is Here: How We Put OpenAI's Speech-to-Speech Model Inside Our Voice AI Agents

If you have ever called a business and known within two seconds that you were talking to a machine, you already understand the problem gpt-realtime was built to solve. The tell was never intelligence. It was timing — that half-beat of dead air while your words were transcribed, shipped to a language model, turned back into text, and re-synthesized into speech. Every layer added delay, and delay is what makes a voice sound artificial.

OpenAI's Realtime API and its gpt-realtime family of models eliminate the relay entirely. One model listens to raw audio and responds with raw audio, in a single conversational stream 1^[2]. We have spent the past stretch integrating it into the voice AI agents we deploy for clients, and we want to walk through what it is, why it matters, and what it actually took to make it production-ready for a real business phone line.

Click to expand

The old voice AI stack was a game of telephone between three different models. gpt-realtime is one model that simply hears you and talks back.

The old pipeline, and why it always sounded like one

Traditional voice agents are a chain: a speech-to-text model transcribes the caller, a language model reasons over the transcript, and a text-to-speech model reads the answer aloud. Three models, three hops, three places to lose information [4].

The latency was the obvious cost, but the subtler one was everything the transcript threw away. Hesitation, tone, a caller talking over the agent, laughter, frustration — none of it survives transcription. The language model reasoned about a flattened version of the conversation, which is why pipeline agents interrupt badly, miss sarcasm, and plow ahead when a caller is clearly confused.

Click to expand

Speech-to-speech models change the physics. Because gpt-realtime processes audio natively, it hears how something was said, not just what was said. It can respond to a change in tone, yield gracefully when interrupted mid-sentence, and match the caller's pacing — the things humans do without thinking and pipelines could never quite fake 1^[2].

What the Realtime API actually gives you

For the technically curious, the shape of the integration is refreshingly simple. Your server mints an ephemeral key, the client opens a WebRTC or WebSocket connection to the realtime endpoint, and audio flows both ways over that single session [2]. For phone calls specifically, the API supports SIP, which is how a realtime model ends up answering an actual business phone number rather than a browser demo 2^[5].

The production model, gpt-realtime-2.1 at the time of writing, supports the pieces a working agent needs: function calling for taking real actions mid-conversation, configurable latency and reasoning behavior, and instructions that persist across the whole session [3]. OpenAI's Agents SDK wraps this in a RealtimeAgent abstraction — define the agent, hand it tools, connect the session [3]. There are even specialized variants for live translation and transcription workflows [2].

How we integrated it at Intueo

Here is the part the launch posts do not cover: a great voice model is the mouth and ears of an agent, not the agent itself. A business phone line needs an agent that knows the business, remembers its callers, and can actually do things — and none of that ships in the box.

So our integration treats gpt-realtime as the conversational layer of a larger stateful agent. When a call comes in over SIP, the realtime session connects, and our agent supplies everything around it: the business's knowledge, the caller's history pulled from the agent's own memory, and a toolbelt of function calls wired into the client's real systems — booking slots on the calendar, looking up an order, updating the CRM, escalating to a human with full context 3^[5].

Click to expand

The memory piece matters most, and it is where our existing architecture paid off. Because our agents are stateful by design — something we wrote about when we fixed our AI receptionist's amnesia — the realtime session is not a blank slate. A repeat caller gets an agent that already knows their last conversation, and whatever happens on today's call is written back to the agent's memory when it ends. gpt-realtime made the conversation feel human. Statefulness makes the relationship feel human.

What changed on real calls

The upgrade is audible in the first exchange. Callers interrupt naturally and the agent stops talking — actually stops, instead of finishing its paragraph. Background noise no longer derails transcription because there is no transcription to derail. And the agent picks up on hesitation: when a caller sounds unsure, it slows down and offers to clarify rather than barreling toward a booking.

The operational wins are quieter but just as real. Fewer abandoned calls in the first ten seconds, which is where robotic-sounding agents lose people. Smoother handoffs, because the agent can keep talking naturally while a function call runs in the background. And a noticeable drop in the "can I just talk to a person?" escape hatch — not because we hid it, but because fewer callers feel the need to reach for it.

The honest caveats

We would be selling you something if we pretended it was all upside. Realtime audio sessions cost more than text pipelines, so per-minute economics matter and we engineer around them. Voice-native models introduce new failure modes — an agent that sounds completely human raises the bar for disclosure, which is why every agent we deploy identifies itself as an AI up front, in line with the rules we covered in our piece on AI voice agent disclosure requirements. And guardrails need rethinking when output is audio: you cannot regex a waveform, so behavioral constraints have to live in the agent layer, not a post-processing filter.

None of these are reasons to wait. They are reasons the model alone is not the product — the agent architecture around it is.

Where this goes

Speech-to-speech is not a feature release; it is the moment voice AI crossed from "impressive demo" to "infrastructure a business can answer its phone with." The models will keep improving and, notably, our architecture is model-agnostic — the same stateful agent that speaks through gpt-realtime today can speak through whatever is best next quarter, without losing a day of its memory.

If your business still routes callers through hold music and phone trees, this is the technology that replaces them. Talk to us and we will show you what your phone line sounds like with a realtime agent answering it.