Speech synthesis is one of the few corners of machine learning where thirty years of progress fits into twenty seconds of listening. Your ear does the evaluation; no benchmark required.
The through-line is one idea: every era of text-to-speech moved the unit of generation down a level of abstraction — from rules about how a voice resonates, to recorded fragments of speech, to statistical parameters, to raw audio samples, to discrete tokens. Each move bought naturalness and gave up control.
The old engines still install, so this is no museum tour. Every era below has a player, and four of the five clips were made on a laptop while writing this.
The map
Five eras, each defined by what it generates. Click one.
1990s · Rules and fragments
Formant synthesis generates speech from hand-written rules, storing no recorded audio at all. Diphone concatenation splices together one recorded copy of each sound-to-sound transition. Intelligible, unmistakably synthetic.
The 1990s: you could understand it, but never mistake it for a person
The first era stored no speech at all. Formant synthesis builds a voice from oscillators and filters tuned to reproduce formants — the resonant frequency bands your throat and mouth impose on a sound, and what makes “ee” differ from “ah”. Rules convert letters to phonemes, phonemes to filter settings. That is how DECtalk spoke, and why Stephen Hawking’s voice sounded as it did.
The alternative: record a human once, then cut the recording up. Diphone concatenation stores one copy of every sound-to-sound transition — the milliseconds spanning “k” into “æ” — then splices and pitch-shifts them into a sentence. Edinburgh’s Festival, released in 1996 and still the ancestor of most open-source TTS, did exactly this.
What changed: arbitrary text became speakable on a machine with almost no memory. The cost was humanity: every syllable is identical every time, because only one copy exists.
The 2000s: two opposite fixes for the same robotic seam
Both 1990s techniques failed at the joins. The next decade produced two opposite responses.
Unit selection attacked it with volume: record hours of one speaker, then search that database at synthesis time for the longest contiguous stretch of real audio matching the target. When the database contains your phrase, the output is a human recording. When it does not, the voice lurches between registers mid-sentence.
Statistical parametric synthesis stored no waveforms at all. The HTS toolkit modelled speech with hidden Markov models as smoothly varying parameters — spectral envelope, pitch, duration — then rebuilt audio with a vocoder, a component that reconstructs a waveform from that compact description. Tiny and easy to bend: change a parameter, change the speaker. It also sounded muffled and buzzy, because averaging is what a statistical model does.
What changed: the field split into hoarding real audio versus modelling its parameters, and neither got both naturalness and flexibility. The neural era broke that deadlock.
The 2010s: the hand-built pipeline collapsed into two networks
DeepMind’s WaveNet (2016) dropped the intermediate representation entirely and predicted the raw waveform one sample at a time, each conditioned on every sample before it — an autoregressive model, in the same sense as a language model predicting the next word. At 16,000-odd predictions per second of audio it was glacially slow, and it beat everything before it.
Tacotron (2017) removed the other half of the stack: a sequence-to-sequence network mapped characters directly to a mel spectrogram, learning from data the pronunciation and prosody rules people had hand-written for decades. Tacotron 2 bolted the two together — a spectrogram predictor feeding WaveNet as a neural vocoder — and scored 4.53 on a 5-point naturalness rating against 4.58 for professionally recorded human speech.
What changed: the linguistic front-end stopped being engineered and started being learned. The price was speed; WaveGlow (2018) began clawing it back with a flow that generates samples in parallel.
The early 2020s: dropping autoregression made it fast enough to ship
Quality was solved; throughput was not. FastSpeech 2 (2020) predicts each phoneme’s duration explicitly, then emits the entire spectrogram in one parallel pass instead of left to right. HiFi-GAN (2020) did the same for the vocoder, training a generator against several discriminators to run hundreds of times faster than real time on a GPU — and, crucially, faster than real time on a laptop CPU. VITS (2021) closed the remaining seam, training both stages end-to-end with no separate vocoder.
What changed: open-source TTS became deployable rather than demonstrable. The limitation was rigidity — a model gave you the voices it was trained on, and a new voice meant a new training run.
Now: speech is just another token stream
The current era rests on the neural audio codec: a model that compresses a waveform into a short sequence of discrete symbols — codec tokens — and decodes them back to sound. Think of it as a vocabulary for audio, the role subword tokens play for text.
Once speech is a token sequence, generating it is a language-modelling problem and the whole transformer toolkit transfers. Chatterbox is explicitly a Llama-style backbone predicting speech tokens; CosyVoice 2, OpenAudio S1-mini and IndexTTS-2 are variations on the theme.
What changed: voice stopped being a property of the weights and became part of the prompt. Give one a few seconds of reference audio and it clones a speaker with no fine-tuning; ask for an emotion and it obliges. The costs return too: autoregression brings back latency and occasional garbled output, and effortless cloning is a real misuse risk — which is why models like Chatterbox watermark what they emit.
Hear thirty years of the same sentence
Here is that arc as sound. Four of the five clips were generated locally by this post’s build script, all saying one sentence so the only variable is technique. These are living descendants, not original binaries — eSpeak NG still synthesises by formant rule, and Flite still ships a diphone database and a parametric voice.
No recorded human speech anywhere in this file. Every sound is rules driving filters.
cmu_us_kal
A real person’s voice, chopped into transitions and pitch-shifted back together. Listen for the seams.
cmu_us_slt Clustergen
Smooth where the diphone voice was jagged, muffled where it was sharp. Clustergen is the same family as HMM-based HTS, not literally HTS.
The one clip not generated here, and a different sentence: Tacotron 2’s checkpoint was never released, so this is mirrored from Google’s demo page.
af_heart
82 million parameters, generated on a laptop CPU in seconds. Try other voices in the official Kokoro Space.
The jump from the third clip to the fifth is the entire neural revolution — the one place where reading about mean opinion scores tells you less than four seconds of listening.
What you can actually download today
All five run locally, and all five carry a permissive licence.
| Model | Provider | Parameters | Underlying tech | Licence |
|---|---|---|---|---|
| Kokoro-82M | hexgrad | 82M | Non-autoregressive StyleTTS 2 architecture with an ISTFTNet decoder | Apache 2.0 |
| Chatterbox / Turbo | Resemble AI | 500M / 350M | Llama-style token generator plus a distilled diffusion decoder | MIT |
| CosyVoice 2 | Alibaba (FunAudioLLM) | 0.5B | LLM-based codec TTS, streaming-optimised to ~150 ms latency | Apache 2.0 |
| OpenAudio S1-mini | Fish Audio | 0.5B | Dual autoregressive (Dual-AR) codec architecture with GFSQ | Apache 2.0 |
| IndexTTS-2 | Bilibili | not published | Autoregressive text-to-semantic then semantic-to-mel, with duration and emotion control | Apache 2.0 |
Caveat: parameter count is a poor proxy for quality here
That table invites a comparison it cannot support. Kokoro is roughly six times smaller than the codec models beside it and still ranks near the top of community listening arenas, because it does a narrower job — fixed voices, no cloning, no emotion prompt. The larger models buy controllability, not better sentences. Pick on the axis you need, and treat naturalness scores as noisy — they shift with the listener pool, the text, and who ran the evaluation.
Where this is heading
The direction of travel is that TTS stops being a separate component. Once speech is tokens, there is no principled reason to generate text and then read it aloud — one model can emit speech tokens directly, removing the prosody errors that come from a synthesiser guessing at intent it never saw. Expect streaming by default rather than as an optimisation, emotion and style addressed in the prompt like everything else, and watermarking to get far more attention, because a technology that clones a voice from six seconds of audio has already outrun the norms around it.
Sources
- WaveNet — arXiv:1609.03499; Tacotron — arXiv:1703.10135; Tacotron 2, including the 4.53 vs 4.58 MOS figures — arXiv:1712.05884; WaveGlow — arXiv:1811.00002
- FastSpeech 2 — arXiv:2006.04558; HiFi-GAN — arXiv:2010.05646; VITS — arXiv:2106.06103
- Festival (CSTR Edinburgh), Flite and HTS for the pre-neural eras; eSpeak NG for formant synthesis
- Model cards and repositories linked in the table above; Chatterbox parameter counts and PerTh watermarking from Resemble AI
- Tacotron 2 sample clip mirrored from Google’s demo page