Cascade or Native? What the Voice AI Stack Actually Looks Like in 2026

Two architectures, four benchmarks, and the one property that still separates them

Native speech-to-speech models are not merely level with cascaded pipelines on latency — they are ahead, by seconds. What open native models have not closed is the reasoning gap. And what the cascade still sells you is a transcript you can read before the user hears anything, bought at a price the benchmarks make uncomfortably clear.
AI
Voice
Speech
Author

Ravi Kalia

Published

August 9, 2026

Cascade or Native? The Voice AI Stack, August 2026

This post compares the two ways voice agents get built in 2026 — cascade and native speech-to-speech — against four third-party benchmarks, and names the one property that still separates them.

Voice agents in 2026 are built two ways: cascade (audio → text → LLM → text → audio) or native speech-to-speech (audio in, audio out, no inspectable transcript in the middle).

The benchmarks are third-party and dated August 2026. Latency favors native models by seconds on multi-step tasks. The main architectural trade is transcript inspectability vs full-duplex conversation and self-correction handling.

1 Cascade architecture

A cascade (chained pipeline) is three models in sequence:

  • STT (speech-to-text, also called ASR, automatic speech recognition) turns the incoming audio into a string.
  • An LLM (large language model) reads that string and decides what to say and which tools to call.
  • TTS (text-to-speech) turns the reply back into audio.

VAD (voice activity detection) or endpointing decides when the user stopped talking so STT output can finalize.

Properties:

  • Transcript is a real string — log, redact, classify, block, swap LLM/TTS independently, audit later.
  • Transcript is an early commitment: once endpointing fires, audio (hesitations, corrections, tone) is discarded; downstream sees only the string.

2 Native speech-to-speech architecture

Native S2S — one model: audio in, audio out; no grep-able transcript in the middle.

Mechanism:

  • Neural audio codec compresses waveform → discrete speech tokens.
  • Model predicts more tokens; decoder → waveform.
  • No sentence-level endpoint required before processing.

Full duplex — listen and speak concurrently; enables interruption and backchanneling (“mm-hmm”).

%%{init: {'theme':'base', 'themeVariables': {'primaryColor':'#ffffff','primaryBorderColor':'#4A3AA7','primaryTextColor':'#1a1a1a','lineColor':'#4A3AA7','edgeLabelBackground':'#ffffff'}}}%%
flowchart LR
  subgraph C["Cascaded pipeline"]
    direction LR
    A1["🎤 audio in"] --> V["VAD /<br/>endpointing"]
    V --> S["STT"]
    S -->|"transcript<br/>(inspectable, committed)"| L["LLM"]
    L -->|"reply text"| T["TTS"]
    T --> O1["🔊 audio out"]
  end

  subgraph N["Native speech-to-speech"]
    direction LR
    A2["🎤 audio in"] --> E["audio codec<br/>→ speech tokens"]
    E --> M["single S2S model<br/>(listens + speaks at once)"]
    M --> D["speech tokens<br/>→ waveform"]
    D --> O2["🔊 audio out"]
  end

  %% Invisible link, purely to pin the cascade above the native path so the
  %% diagram reads in the same order as the prose.
  C ~~~ N

2.1 Native model variants

Two designs dominate benchmark results:

Speech-token models — trained primarily on audio tokens; text is ancillary. Moshi (Kyutai): 7B text backbone + Mimi codec at 12.5 Hz (~80 ms per token); reported 160 ms theoretical / 200 ms practical latency.1 PersonaPlex (NVIDIA) extends Moshi/Moshiko with role and voice conditioning.2 Strong conversational timing; weak reasoning on audio benchmarks.

Text-LLM-with-speech-I/O — general text LLM with audio encoder and TTS decoder. Nemotron 3 VoiceChat (3 August 2026): Fast Conformer encoder → Nemotron Nano v2 9B → text tokens + TTS + tool-call channel.3 Native full duplex with internal text bottleneck.

Cascade vs native is a spectrum of how early and irreversibly audio becomes text.

3 Benchmark metrics

Quoted latency numbers often measure different quantities. Definitions:

  • TTFA (time to first audio) — seconds from the user finishing to the first sound from the agent. Note that “first sound” is often a filler (“let me check that”), which is why it can diverge wildly from when useful information arrives.
  • Task completion latency — seconds until the agent says the thing you were waiting for. Much more honest, much less often quoted.
  • RTFx (inverse real-time factor) — how many seconds of audio a model processes per second of compute, so above 1 is faster than real time and higher is better. Its reciprocal, RTF, runs the other way: below 1 is faster than real time. Only RTFx is quoted in this post.
  • WER (word error rate) — the fraction of words a transcriber gets wrong, as insertions, deletions and substitutions, over the reference transcript. Lower is better; it is a percentage but not a percentage correct.
  • Task adherence / turn-taking — does the agent take its turn when it should, stay quiet during your pauses, and stop when you interrupt?
  • Elo / win-rate — for TTS, listeners hear two clips of the same sentence and pick the better one; ratings are fit to those pairwise votes. It measures preference, not accuracy, and it moves week to week.

3.1 Benchmark sources

Four sources; each built by a party with stated motives:

Open ASR Leaderboard (Hugging Face) — reproducible WER on public corpora: AMI, Earnings22, LibriSpeech (clean/other), SPGISpeech, VoxPopuli, GigaSpeech. Average WER = macro-average over those seven sets after text normalization (verified by reproducing averages for three models). Speed on one NVIDIA H200 per model.4

Full-Duplex-Bench-v3 (National Taiwan University + NVIDIA) — turn-taking and tool use on disfluent speech: 100 recordings, 12 speakers, built-in mics, real background noise, five disfluency categories, multi-step tool scenarios.5

Artificial Analysis — commercial Speech-to-Speech Index: Big Bench Audio (1,000 BBH questions read aloud) + Full-Duplex-Bench subset + τ-Voice (customer-service tool task). Speech Arena for TTS pairwise preference.6(https://artificialanalysis.ai/methodology/speech-to-speech-benchmarking); leaderboard read 9 August 2026.]

Objective: mishearing, turn-taking errors, task failure rates for deployment decisions (pharmacy, insurance, emergency triage). Wrong answers can cause real harm.

Self-reported vendor numbers are flagged; third-party evaluation preferred.

4 Latency comparison

Full-Duplex-Bench-v3: six configurations, identical audio, one LiveKit harness, one server region. Five native S2S; one cascade (Whisper → GPT-4o → TTS).

Latency on multi-step tool tasks with real, disfluent speech. Source: Full-Duplex-Bench-v3, Table 6.
System First word (s) Task completion (s)
Ultravox v0.7 (open weights) 3.88 8.40
Gemini Live 3.1 3.95 4.25
Grok 5.97 6.65
GPT-Realtime 6.36 6.89
Gemini Live 2.5 7.03 7.26
Cascaded (Whisper→GPT-4o→TTS) 8.78 10.12

Cascade is slowest on both metrics; task completion ~2.4× fastest native (Gemini Live 3.1).

Caveats:

  • Gemini Live 3.1 average excludes 22% of runs where it produced no speech; cascade responded in 100%.
  • Ultravox: fastest first word, slowest completion; filler sentences in 88% of scenarios before tool calls.
  • Numbers include multi-step tool execution on disfluent speech — not comparable to Moshi’s ~200 ms streaming latency.

5 Metric comparability

Latency figures across sources measure different things:

Moshi’s 200 ms is a theoretical-plus-practical framing latency for the model’s own streaming loop. Artificial Analysis’s TTFA is seconds to the first audio token on a reasoning question over a network. Full-Duplex-Bench-v3’s 4–10 s is task completion including tool calls on real human speech. PersonaPlex’s model card reports 0.170 s for smooth turn-taking on Full-Duplex-Bench. These differ by nearly two orders of magnitude and every one of them is honest about what it measured.

There is a further wrinkle worth flagging: PersonaPlex’s own model card and its paper report different Full-Duplex-Bench figures for the same model,7 which is a small reminder that even primary sources disagree with themselves. Where I could not resolve a discrepancy I have said so rather than picking the flattering number.

Compare within one benchmark only. Charts in this post are internally consistent, not cross-comparable.

6 ASR quality

Cascade value proposition includes transcription. Open ASR Leaderboard runs open and commercial models on identical corpora.

WER = fraction of words wrong (insertions + deletions + substitutions). 4.51 ≈ one word in 22 wrong.

Average word error rate across the seven English short-form test sets. Source: Hugging Face Open ASR Leaderboard, 9 August 2026.
Model Avg WER % Weights
Microsoft Azure Speech (06-2026) 4.51 API
ElevenLabs Scribe v2 4.65 API
IBM Granite Speech 4.1 2B 4.90 Open
Qwen3-ASR 1.7B 5.02 Open
AssemblyAI Universal-3.5 Pro 5.03 API
NVIDIA Canary-Qwen 2.5B 5.06 Open
Cohere Transcribe (03-2026) 5.20 Open
NVIDIA Parakeet TDT 0.6B v2 5.39 Open
Kyutai STT 2.6B 5.74 Open
Speechmatics Enhanced 5.90 API
OpenAI Whisper large-v3 6.55 Open

Field spans ~2 WER points; open weights interleaved with APIs. Whisper (6.55) trails this group.

Caveats:

  • Unclear whether commercial API rows are maintainer-run or vendor-submitted on public splits.
  • English short-form only; noisy mobile calls with accent differ from curated corpora.

7 TTS quality and pricing

Artificial Analysis Speech Arena: blind pairwise preference → Elo ratings with confidence intervals.

Text-to-speech listening preference, Artificial Analysis Speech Arena, 9 August 2026. Bars are 95% confidence intervals.
Model Elo 95% CI Weights List price / 1M chars
Qwen-Audio-3.0-TTS-Plus 1,229 ±15 Proprietary $27.60
Gemini 3.1 Flash TTS 1,210 ±13 Proprietary $18.30
Cartesia Sonic 3.5 1,203 ±13 Proprietary $49.00
Inworld Realtime TTS 1.5 Max 1,194 ±13 Proprietary $26.00
ElevenLabs Eleven v3 1,171 ±12 Proprietary $100.00
Fish Audio S2 Pro 1,121 ±14 Open $15.00
StepFun Step Audio EditX 1,109 ±14 Open —
OpenAI TTS-1 HD 1,097 ±12 Proprietary $30.00
Mistral Voxtral TTS 1,067 ±14 Open $16.00
NVIDIA Magpie-Multilingual 357M 1,065 ±14 Open —
Kokoro 82M v1.0 1,056 ±11 Open $0.65 †
Resemble Chatterbox 1,014 ±12 Open $25.00
Coqui XTTS v2 914 ±15 Open $40.40 † ‡

Notes:

  • Qwen leads; ElevenLabs (5th) costs 3.8–5.5× models above it on quality; Elo intervals for Inworld vs Eleven v3 overlap.
  • † Third-party hosting (e.g. Kokoro via Replicate), not author list price.
  • ‡ Coqui shut down early 2024; community fork only.
  • Best open TTS trails best proprietary by ~108 Elo — voice selection and prompt tuning often matter more.

TTS is largely commoditized; product differentiation on voice quality alone is weak.

8 Reasoning vs conversational dynamics

Artificial Analysis scores native S2S on two axes: - Speech reasoning — Big Bench Audio (hard questions as audio). - Conversational dynamics — Full-Duplex-Bench turn-taking.

Native speech-to-speech models, reasoning against conversational dynamics. Source: Artificial Analysis, 9 August 2026.
Model Speech reasoning % Conv. dynamics % Weights
Qwen Audio 3.0 Realtime Plus 99 98.4 Proprietary
Grok Voice Think Fast 2.0 97 95.1 Proprietary
Gemini 3.1 Flash (High) 97 74.3 Proprietary
GPT-Realtime-2.1 High 96 95.7 Proprietary
Deepslate Opal 85 85.7 Proprietary
Qwen3 Omni Flash 59 72.7 Proprietary
Freeze-Omni 33 58.7 Open
Nemotron 3 VoiceChat 27 52.9 Open
PersonaPlex 19 91.0 Open
FLM-Audio 16 62.0 Open
Moshi 4 61.0 Open

Rows sorted by speech reasoning; ties by conversational dynamics. Deepslate: unverified beyond leaderboard rows.

Open models: 4–33% speech reasoning vs 96–99% proprietary frontier. Moshi: 4% on audio reasoning. PersonaPlex: 91.0% conversational dynamics (highest open).

Speech-token training optimizes next-frame prediction (~12.5 Hz); weak multi-step deduction signal.

No open-weights model has full Speech-to-Speech Index score (missing τ-Voice agentic component).

8.1 Big Bench Audio caveat

Big Bench Audio = Big Bench Hard read aloud (23 voices: OpenAI, Azure, Amazon Polly).8 Clean synthetic speech; tests text-shaped reasoning via audio, not noisy telephony.

9 Cascade self-correction failure

Full-Duplex-Bench-v3: cascaded Whisper→GPT-4o→TTS vs GPT-Realtime (both GPT-4o class LLM).

  • Task completion: cascade 45.0% vs GPT-Realtime 60.0%.
  • Self-corrections (user changes mind mid-utterance): cascade 17.6% vs GPT-Realtime 58.8%.

Whisper finalizes transcription before correction arrives; downstream never sees the change. Transcript is lossy, early, irreversible — same property that enables logging/redaction.

Cascade: 100% turn-take rate (only system). Gemini Live 3.1: silent in 22% of scenarios (86% of those had executed tools correctly but never spoke).

10 Vendor landscape

August 2026 vendors by layers shipped (STT, LLM, TTS, cascade, native S2S). Rows verified against company sites or model cards.

Company / project Type Layers Notable models Licence / pricing Best known for
OpenAI Closed API STT, LLM, TTS, native S2S gpt-realtime-2.1, gpt-4o-transcribe, Whisper $32 / $64 per 1M audio in/out tokens; Whisper weights MIT The default realtime API; strong tool use
Google Closed API Native S2S, TTS, STT Gemini 3.1 Flash Live, Gemini 3.1 Flash TTS $3 / $12 per 1M audio in/out tokens Cheapest frontier native audio; fast
Microsoft / Azure Closed API STT, TTS Azure Speech, Azure HD 2.5, MAI-Voice-1 Per-hour enterprise pricing Top of the ASR leaderboard; enterprise reach
Alibaba / Qwen Mixed Native S2S, STT, TTS Qwen Audio 3.0 Realtime Plus, Qwen3-Omni, Qwen3-ASR Qwen3-Omni Apache 2.0; 3.5 generation API-only Current frontier on both S2S axes
xAI Closed API Native S2S, TTS Grok Voice Think Fast 2.0 Per-token Fastest TTFA among frontier native models
Fixie Open weights Native S2S Ultravox v0.7 Open weights Open tool-use S2S without a separate ASR stage
Mistral Mixed TTS, STT Voxtral TTS, Voxtral Mini/Small Open weights (Apache 2.0) Open TTS and STT from a frontier lab
NVIDIA Open weights Native S2S, STT, TTS Nemotron 3 VoiceChat 11B, PersonaPlex 7B, Canary, Parakeet, Magpie OpenMDW 1.1 / NVIDIA Open Model License / CC-BY-4.0 The open full-duplex frontier
Kyutai Open weights (nonprofit) Native S2S, STT, TTS Moshi, Kyutai STT/TTS, Pocket TTS Weights CC-BY-4.0; code MIT / Apache Proved full-duplex was possible
ElevenLabs Closed API TTS, STT, agents Eleven v3, Scribe v2 TTS $100 / 1M chars; Scribe v2 $0.22/hr; Agents $0.08/min Voice quality brand; large voice library
Deepgram Closed API STT, TTS, agents Nova-3, Flux, Aura-2 STT from $0.0048/min; Voice Agent $0.075/min Cheap, fast streaming STT for telephony
AssemblyAI Closed API STT, agents Universal-3.5 Pro $0.21/hr async, $0.45/hr realtime; agents $4.50/hr Developer-friendly STT with rich add-ons
Cartesia Closed API TTS, STT, agents Sonic 3.5, Ink-2, Line Plan-based (≈$49/1M chars normalised); agents $0.06/min Very low-latency TTS
Rime Closed API TTS Arcana v3, Coda, Mist V2 $50 / 1M chars Telephony-tuned conversational voices
Fish Audio Mixed TTS S2.1 Pro, S2 Pro, OpenAudio S1 S2 Pro open weights; API $15 / 1M chars Best open-weights TTS on the arena
Resemble AI Mixed TTS Chatterbox, Chatterbox HD / Turbo / Multilingual Chatterbox MIT Permissive open TTS with voice cloning
Inworld Closed API TTS Realtime TTS 1.5 Max, TTS-2 $26 / 1M chars Top-tier arena quality at mid price
Pipecat Open source Orchestration — BSD-2, maintained by Daily Composable pipeline; bring your own stack
LiveKit Open source + cloud Transport + orchestration LiveKit Agents Open core, hosted cloud WebRTC-native infra; SIP telephony
Vapi Platform Full stack — Per-minute Fastest path to a working phone agent
Retell AI Platform Full stack — Per-minute Contact-centre replacement, multi-channel

Coqui (XTTS v2) company closed early 2024; weights remain on leaderboard.

11 Deployment recommendations

Layer quality differences within cascade are small; licensing and latency often dominate.

Cascade stack:

  • STT, open: NVIDIA Parakeet TDT 0.6B v2 — 5.39 WER at an RTFx of 6,038,9 by far the best accuracy-per-unit-compute here, under a CC-BY-4.0 licence that lets you ship it commercially with attribution. Choose IBM Granite Speech 4.1 2B (4.90) if you want the accuracy and can pay roughly 11× the compute per second of audio.
  • STT, paid: Deepgram for telephony, on price and streaming maturity rather than raw WER; Azure Speech if you want the lowest error rate on the board and are already in that ecosystem.
  • LLM: whatever you already trust for tool calling. This is the one stage where voice imposes no special requirement beyond time-to-first-token.
  • TTS, open: Fish Audio S2 Pro for quality, Kokoro 82M if you need it to run anywhere — it is within 65 Elo of S2 Pro at a fraction of the footprint.
  • TTS, paid: Cartesia Sonic 3.5 or Gemini 3.1 Flash TTS. Both sit in the top five and both rank above Eleven v3, at half and a fifth of its price respectively.

Native stack:

  • Closed: GPT-Realtime if tool use matters — it leads Full-Duplex-Bench-v3 on task success, finishing 60.0% of scenarios on the first attempt, and on self-correction handling, though not on latency, where it is fourth of six. Note that the benchmark’s own footnote points at gpt-realtime-1.5, so that result is not evidence about the 2.1 generation; the 2.1 numbers in this post come from Artificial Analysis instead. Qwen Audio 3.0 Realtime Plus tops both Artificial Analysis axes at a fraction of the cost, with the caveat that this is a benchmark lead I have seen independently confirmed only by Artificial Analysis, and that data residency — which country the audio is legally allowed to be processed in — will decide it for many buyers before quality does. Gemini 3.1 Flash Live is the value option if you can tolerate its silence failures.
  • Open: Nemotron 3 VoiceChat 11B. It is the only open full-duplex model with live tool calling, and its text-LLM-in-the-middle design is why it reasons better than every pure speech-token model on the board — 27 against PersonaPlex’s 19, FLM-Audio’s 16 and Moshi’s 4. It is not the top open reasoner outright: Freeze-Omni scores 33, from the same architectural family, but without the tool calling. Reach for PersonaPlex instead only if conversational feel is the product and reasoning genuinely is not.

Open native gaps: reasoning, agentic tool evaluation, production maturity. Caught up: conversational timing (PersonaPlex), TTS quality.

Typical choice: short/scripted calls → native; long/regulated/tool-heavy → cascade for inspectability and guaranteed response.

12 Outlook

Opinion, not measurement:

Hybrids win, and they are already here (fairly likely). Nemotron 3 VoiceChat is a full-duplex native model that predicts text tokens internally and routes tool calls through a dedicated channel. That is the synthesis: a text representation you can read and act on, produced without the endpointing commitment that makes cascades slow and brittle on corrections. I expect the “cascade versus native” framing to look dated within a year, replaced by arguments about where in the model the text channel sits and what you are allowed to do with it.

TTS differentiation moves to latency, cost and control (fairly likely). A 150× price spread across a field where the bottom is still preferred a third of the time is not a stable market. Quality is close to saturated at the top; what is left to compete on is time-to-first-byte, price per minute, and controllability — emotion, pacing, pronunciation of names and drug names.

Consolidation among mid-tier voice API vendors (moderately likely). There are more companies selling near-identical STT and TTS endpoints than the margins support, and the frontier labs now ship all three layers themselves at prices the specialists struggle to match. The survivors will be those who own a vertical or the telephony path, not those with the prettiest voice.

Telephony and enterprise integration is the durable moat (fairly likely). The plumbing that connects a model to an actual phone number is unglamorous, slow to build, and not commoditising at anything like the rate the models are: SIP trunking, the protocol that carries calls between networks; routing them to the right queue; recording them in a form a compliance team will accept; and writing the outcome back into whichever customer-records system the business already runs on. This is LiveKit’s and Deepgram’s structural advantage, and the reason platforms like Vapi and Retell can sell against raw APIs.

Open models close the reasoning gap (speculative, and the one I am least sure of). The gap is currently enormous. But the mechanism is now understood — put a real language model in the middle and the reasoning follows — and Nemotron 3 VoiceChat is a proof of concept six days old. If the open frontier converges on that recipe rather than on pure speech-token models, the gap could close much faster than the current numbers suggest. If it does not, open native audio stays a research curiosity with excellent manners.

13 Architecture choice

Primary trade-off is transcript (when produced, whether revocable), not raw speed.

  • Cascade: inspectable string; seconds of latency; poor self-correction when endpointing commits early.
  • Native: full duplex and interruption; no grep-able middle; frontier proprietary models strong on both axes; open native strong on timing only.

Selection:

  • Cascade when transcript is the product (audit, regulation, long tool chains).
  • Native when conversation feel matters and tasks are short.
  • Hybrids (text channel inside native model) emerging.

Scope limit: English, short-form, benchmark audio; noisy mobile multilingual calls differ.

14 Data accuracy notes

Figures read from primary sources (papers, model cards, live leaderboards, pricing pages) on 9 August 2026. Secondary aggregator posts omitted when contradicted by live data.

Unresolved / flagged inline:

  • Commercial API evaluation supervision on Open ASR public splits.
  • PersonaPlex model-card vs paper Full-Duplex-Bench configuration mismatch.
  • No open-weights latency on same harness as proprietary TTFA.

Leaderboards change weekly; verify live before deciding.

Native. Wins. Latency. Cascade. Keeps. The. Transcript. Reasoning. Still. Decides.

15 References

Papers and benchmarks

Models

Pricing pages (read 9 August 2026)

Frameworks

Footnotes

  1. Défossez et al., Moshi: a speech-text foundation model for real-time dialogue, arXiv:2410.00037. Latency figures from the abstract; codec details from §3.3.1. Self-reported by the model’s authors.↩︎

  2. nvidia/personaplex-7b-v1 model card (released 15 January 2026, NVIDIA Open Model License; code MIT) and Roy et al., PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models, arXiv:2602.06053. I have deliberately not given a submission date for the preprint: the listing I read reported 14 January 2026, which is inconsistent with an identifier encoding February, and I could not resolve which is right.↩︎

  3. nvidia/NVIDIA-NemotronLabs-VoiceChat-11B, 11B parameters, released under the OpenMDW License v1.1. Architecture description from the model card.↩︎

  4. Srivastav et al., Open ASR Leaderboard, arXiv:2510.06961; harness and hardware details from the project README. Results read from the live hf-audio/open-asr-leaderboard-results dataset on 9 August 2026.↩︎

  5. Lin, Chen, Chen and Lee, Full-Duplex-Bench-v3, arXiv:2604.04847. Dataset construction from §3; latency from Table 6.↩︎

  6. Artificial Analysis speech-to-speech benchmarking methodology↩︎

  7. The model card lists smooth turn-taking success 0.908 and interruption success 0.950; the paper’s Table 2 lists 0.992 and 1.000. I could not determine from the published material which configuration each corresponds to.↩︎

  8. ArtificialAnalysis/big_bench_audio dataset card.↩︎

  9. Same Open ASR Leaderboard results dataset as the word-error table, read 9 August 2026. RTFx there is measured on one H200; IBM Granite Speech 4.1 2B records 547 on the same hardware.↩︎