Text-to-Speech › Neural Vocoders

The mel-spectrogram gives a compact, perceptually relevant representation, but it discards phase information. Reconstructing a waveform requires estimating these missing phases, a problem classical signal processing solves through iterative algorithms that often produce metallic artifacts or temporal smearing. Neural vocoders learn to generate waveforms directly, conditioned on spectrograms, creating materially more natural speech with crisp transients and appropriate noise characteristics.

WaveNet, developed by DeepMind, pioneered autoregressive neural vocoding. Operating directly on the raw audio waveform, WaveNet models the joint probability of audio samples as a product of conditional probabilities:

$$p(x) = \prod_{t=1}^{T} p(x_t | x_1, ..., x_{t-1}, c)$$

where:

• $p(x)$: the joint probability of the audio waveform $x$ consisting of $T$ samples
• $T$: the total number of audio samples in the waveform
• $t$: the current time step index (from 1 to $T$)
• $x_t$: the audio sample at time step $t$
• $x_1, ..., x_{t-1}$: all previous audio samples before time $t$
• $c$: the conditioning information (the mel-spectrogram)
• $p(x_t | x_1, ..., x_{t-1}, c)$: the conditional probability of sample $x_t$ given previous samples and conditioning

This mirrors the autoregressive language modeling we discussed in Part XXII: Causal Language Modeling, but applied to continuous audio samples rather than discrete tokens. The model learns the statistical dependencies between consecutive samples. This keeps the generated waveform is locally consistent and free of discontinuities.

WaveNet uses dilated causal convolutions, which are convolutions with gaps (dilations) that exponentially increase the receptive field while maintaining causality (no access to future samples). A stack of dilated convolutions with dilation rates $1, 2, 4, 8, ..., 512$ allows the model to capture temporal dependencies across thousands of samples (hundreds of milliseconds) using relatively few layers. The causal constraint ensures that the prediction for sample $t$ depends only on samples $1$ through $t-1$, preserving the autoregressive property and preventing the model from "cheating" by looking at future values during training.

The model outputs a categorical distribution over quantized audio amplitudes (typically 8-bit or 16-bit $\mu$-law encoded), sampled autoregressively to generate waveforms. While WaveNet produces excellent quality with rich timbre and natural-sounding speech, its sample-by-sample generation is computationally prohibitive for real-time applications, often requiring specialized hardware like GPUs or TPUs to reach acceptable speeds.

WaveRNN addresses this by using a single-layer recurrent network with dual softmax outputs for coarse and fine quantization levels, reducing computational complexity while maintaining quality. The dual softmax splits the prediction of 16-bit audio into two 8-bit predictions (coarse and fine), simplifying the probability distribution. However, both models remain slower than real-time without specialized hardware acceleration, limiting their deployment in resource-constrained environments.

Generative Adversarial Networks give an alternative path to fast, high-quality vocoding. Rather than modeling the exact likelihood of each audio sample, which requires expensive autoregressive computation, GAN vocoders learn to produce waveforms that are indistinguishable from real speech according to a discriminator network. This adversarial approach trades exact probabilistic modeling for speed and perceptual quality.

MelGAN and HiFi-GAN stand for the state of the art in efficient neural vocoding. These models use transposed convolutions to upsample mel-spectrograms to audio rates, followed by residual blocks with dilated convolutions to model temporal dependencies. The generator operates non-autoregressively, creating audio frames in parallel, which lets real-time synthesis on standard hardware.

Training employs multiple discriminators operating at different time scales:

• A multi-scale discriminator examines the waveform at different resolutions (raw audio, average-pooled by 2, by 4), capturing structure at various temporal granularities
• A multi-period discriminator examines different periodic patterns (every 2nd sample, every 3rd sample, etc.), capturing pitch periodicity and harmonic structure that are important for natural-sounding speech

The adversarial loss encourages the generator to match the distribution of natural speech:

$$\mathcal{L}_{\text{adv}} = \mathbb{E}_{(x,s)}[(D(x) - 1)^2] + \mathbb{E}_s[D(G(s))^2]$$

where:

• $\mathcal{L}_{\text{adv}}$: the adversarial loss function training the generator to fool the discriminator
• $\mathbb{E}_{(x,s)}$: the expectation over real audio $x$ and corresponding spectrogram $s$ pairs from the training data
• $x$: real audio waveform sampled from the data distribution
• $s$: the input mel-spectrogram used as conditioning for generation
• $D$: the discriminator network that classifies audio as real (1) or fake (0)
• $G$: the generator network (vocoder) that synthesizes audio from spectrograms
• $\mathbb{E}_s$: the expectation over generated samples produced from spectrogram $s$

Additional feature matching losses (L1 distance between intermediate discriminator layer activations for real vs. generated audio) and mel-spectrogram reconstruction losses stabilize training and ensure that the generated audio remains faithful to the conditioning spectrogram, preventing mode collapse where the generator produces plausible-sounding audio that does not match the input text.

HiFi-GAN reaches real-time synthesis on CPUs while matching WaveNet quality by using a multi-receptive field fusion (MRF) module that processes audio through parallel dilated convolution branches with different dilation rates, similar to the Inception architecture in computer vision. This allows the model to capture both short-term transients (like consonant bursts) and long-term structure (like pitch periodicity) simultaneously.

Flow-based models like WaveGlow and WaveFlow give an alternative to adversarial training, learning an invertible transformation between a simple noise distribution and the complex distribution of audio waveforms. These models allow exact likelihood computation and efficient sampling through the change of variables formula. Given an invertible function $f$ that turns a latent noise vector $z$ into an audio sample $x = f(z)$, the log-likelihood of $x$ is:

$$\log p(x) = \log p(z) + \log \left| \det \frac{\partial f}{\partial z} \right|$$

where:

• $\log p(x)$: the log-likelihood of the audio waveform $x$ under the model
• $\log p(z)$: the log-likelihood of the latent variable $z$ under the base distribution (typically a standard Gaussian)
• $f$: the learned invertible transformation mapping from latent space $z$ to data space $x$
• $z$: the latent noise variable drawn from a simple base distribution
• $\frac{\partial f}{\partial z}$: the Jacobian matrix of partial derivatives of $f$ with respect to $z$
• $\det \frac{\partial f}{\partial z}$: the determinant of the Jacobian, accounting for the volume change under the transformation

The Jacobian determinant term is what makes normalizing flows computationally tractable. For a general invertible function, computing this determinant requires $O(d^3)$ operations for $d$-dimensional data, which makes it prohibitive for raw audio. Flow models avoid this by constructing $f$ as a composition of simple bijections with triangular or block-diagonal Jacobians, for which the determinant reduces to a product of diagonal elements. WaveGlow uses affine coupling layers where each layer splits the input into two halves, uses one half to predict scale and shift parameters applied to the other half, and the resulting Jacobian is triangular by construction.

Flow-based vocoders train stably and allow exact likelihood evaluation, which is useful for model selection and debugging. They avoid the training instabilities associated with GAN discriminators and the exposure bias of autoregressive models. However, they historically lagged GANs in perceptual quality, though recent architectures have narrowed this gap through improved coupling layers and conditioning mechanisms that incorporate the mel-spectrogram more deeply into each transformation stage.

Diffusion-based vocoders stand for the most recent advancement in this space. Models like DiffWave and WaveGrad treat waveform generation as a denoising problem: starting from Gaussian noise, the model iteratively refines the signal over $T$ diffusion steps, each step guided by the mel-spectrogram conditioning. At each step $t$, the model predicts the noise component $\epsilon$ that was added to the clean signal at that diffusion level, and subtracts an estimate of this noise to take one step toward the clean waveform. Because the model learns a smooth denoising function rather than a single direct mapping, diffusion models can capture complex multimodal distributions in the audio space, creating outputs with rich textural detail. The primary limitation is inference speed: achieving high quality typically requires 50 to 200 diffusion steps, each of which involves a full neural network forward pass. Recent work has focused on distilling diffusion models into fewer steps, using progressive distillation techniques that teach a student model to jump over multiple original steps in a single stride, achieving real-time performance with as few as 4 to 6 inference steps.