ElevenLabs wants to run your voice agent. Here’s what developers give up.

ElevenLabs launched Eleven Realtime on Tuesday, introducing two managed voice-agent models, Mini and Max, that integrate speech recognition, reasoning, turn-taking, and voice generation. Both use a cascaded architecture, but developers can’t select or independently configure the underlying models.

With this launch, the company is effectively taking over the voice-agent harness, including turn detection, interruption handling, and orchestration.

With this launch, the company is effectively taking over the voice-agent harness, including turn detection, interruption handling, and orchestration. That reduces the work required to build and maintain a voice-agent stack, but limits developers’ ability to independently troubleshoot, optimize, or replace its underlying components.

Developers can configure prompts, voices, tools, knowledge bases, and turn-taking settings, including eagerness, timeouts, and interruption handling, through ElevenAgents.

The alternative is Custom mode, which lets developers select the underlying language and speech models and configure the pipeline themselves. Realtime trades that flexibility for a managed implementation. Developers retain control over agent behavior, but not the individual models responsible for it.

Benchmarks from an outside tester

Artificial Analysis generated the performance figures by evaluating both models on speech reasoning, conversational dynamics, and human preference. On Big Bench Audio, a benchmark of 1,000 spoken reasoning questions, Max scored 98.8%, compared with 75.9% for Mini.

ElevenLabs recommends Max for agents that make frequent or complex tool calls.

The gap was even more apparent in tool use. Max completed 87.1% of tasks, while Mini managed 60.3%. ElevenLabs recommends Max for agents that make frequent or complex tool calls. Developers building agents to handle bookings, customer accounts, or other multi-step workflows will want to test that difference for themselves.

The company sets Mini as the default, saying that it is capable of handling most use cases. Mini also performed slightly better on Full-Duplex-Bench, which tests pauses, interruptions, and turn-taking, scoring 95.8% against Max’s 94.1%.

The two models are much closer on latency. Max averaged 570 milliseconds to first audio, compared with 620 milliseconds for Mini. Artificial Analysis’s June Speech-to-Speech Index recorded a faster result of 440 milliseconds for Deepslate Opal, although that earlier benchmark isn’t a direct comparison with Eleven Realtime.

Migrating from OpenAI’s Realtime API

ElevenLabs is also courting teams that already run OpenAI’s Realtime API. The migration guide says Eleven Realtime speaks the same protocol over the same WebSocket transport, so an app built on OpenAI’s SDK can point at ElevenLabs without switching SDKs. The change is the endpoint and the model name.

Migration isn’t free, though. The endpoint supports WebSocket only, so a WebRTC app has to change transports first. The SDK handles audio on its own only over WebRTC. On WebSocket, developers capture and play 24kHz PCM16 audio themselves. A WebRTC app gets that capture and playback from the SDK today, so moving over means building both.

The Realtime API also runs apart from ElevenAgents. Phone and WhatsApp channels, knowledge bases, and monitoring all sit on the platform side. The company says teams that need those features should build on ElevenAgents from the start since a team that begins with the bare API to move quickly may end up rebuilding them later.

Mini versus Max tradeoffs

According to the ElevenAgents documentation, Mini and Max agents can transfer calls between each other, but neither can transfer calls to or from agents running in Custom mode. Mini also lacks support for Zero Retention Mode and HIPAA configurations. Max supports both in the US and EU, although it isn’t available in the India or Singapore data residency environments.

Max costs more per token, but its stronger tool-use performance could make it cheaper for workloads where Mini requires more attempts to finish a task.

Both models are billed at the standard ElevenAgents per-minute rate, plus token charges. Max costs more per token, but its stronger tool-use performance could make it cheaper for workloads where Mini requires more attempts to finish a task.

Testing Before Trusting

ElevenLabs handles the underlying models and orchestration, but developers still need enough diagnostic information to figure out what went wrong when a call fails. It’s unclear how much of that information Realtime makes available.

The benchmarks are promising, but they don’t show how Realtime performs under production load. Developers will need to see whether performance holds up and whether ElevenLabs provides enough information to track down failures.

ElevenLabs did not respond to The New Stack’s request for comment by publication time.

The post ElevenLabs wants to run your voice agent. Here’s what developers give up. appeared first on The New Stack.