Text-to-Speech › End-to-End TTS: Eliminating the Two-Stage Pipeline

The two-stage paradigm, with a separate acoustic model creating spectrograms and a separate vocoder reconstructing waveforms, dominated neural TTS for several years. While effective, this separation introduces a training mismatch: the acoustic model is trained to minimize spectrogram reconstruction error, but the vocoder is trained to produce natural-sounding audio from spectrograms. Any imperfections in the predicted spectrogram, such as slightly blurred formants or missing fine detail in fricative regions, propagate into the vocoder's input at inference time even though the vocoder was never trained to handle such imperfect inputs. This train-test mismatch can cause the vocoder to amplify subtle acoustic model errors into audible artifacts.

End-to-end TTS models sidestep this mismatch by training a single model that maps text directly to waveforms, optimizing a perceptual loss that shows the final audio quality rather than an intermediate spectrogram distance. The most influential end-to-end approach is VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech), which combines a conditional variational autoencoder with adversarial training to produce a model capable of generating high-quality speech from phoneme sequences in a single forward pass.

VITS frames TTS as a conditional variational autoencoder (CVAE) problem. The encoder, called the posterior encoder, takes as input the raw audio waveform (specifically, its linear-scale spectrogram) during training and encodes it into a latent variable $z$ that captures the stochastic aspects of the audio. A normalizing flow, called the prior flow, turns a simple Gaussian conditioned on the text encoder output into a distribution that approximates the posterior. At inference time, the prior flow samples $z$ from the text-conditioned prior, and a HiFi-GAN decoder renders this latent into a waveform. The training objective combines three terms:

1. The reconstruction loss: how well the decoded audio matches the original recording, measured in the mel-spectrogram domain
2. The KL divergence: how closely the prior distribution matches the posterior distribution, enforcing that the text encoder learns to predict what the audio encoder observes
3. The adversarial loss: how convincingly the generated audio fools a multi-period discriminator identical to the one used in HiFi-GAN

The CVAE framework creates a principled probabilistic foundation for the training signal. The KL divergence term forces the text encoder to learn a rich intermediate representation that captures the linguistic content and the prosodic and speaker-specific aspects necessary to reconstruct the waveform. Because both encoder and decoder are trained jointly on real audio, the model learns to generate waveforms that are consistent with natural speech statistics without the train-test mismatch of the two-stage pipeline.

VITS also introduces stochastic duration prediction using a flow-based duration model. Unlike the deterministic duration predictor in FastSpeech 2, which outputs a single duration estimate per phoneme, VITS models duration as a latent variable with a learned prior and posterior. This allows the model to sample different duration values at inference time, creating natural prosodic variation across multiple readings of the same text, rather than always generating identically timed speech. The stochastic nature mirrors how human speakers naturally vary their timing with each utterance while maintaining overall intelligibility.

The practical result is that VITS reaches near-human quality in MOS evaluations while operating as a single unified model that requires no separate vocoder fine-tuning or inference-time alignment. The architecture generalizes naturally to multi-speaker and cross-lingual scenarios by conditioning on speaker embeddings or language tokens, which makes it a versatile foundation for production TTS systems. Facebook's MMS (Massively Multilingual Speech) TTS model, which we use in the code implementation below, builds directly on the VITS framework to support over 1,000 languages with a single model architecture.