Text-to-Speech: Acoustic Models and Neural Vocoders

Text-to-speech systems convert written text into spoken audio, creating a direct interface between written language and acoustic signals. TTS bridges the gap between the symbolic world of text and the acoustic world of sound waves, letting applications from voice assistants and audiobook generation to accessibility tools and language learning platforms. The challenge lies in converting discrete, abstract symbols into continuous physical vibrations that carry lexical meaning and prosody, including emotion and speaker identity. A reading assistant that speaks in a flat, robotic monotone fails users even when every word is correctly pronounced, because naturalness and expressiveness are prerequisites for comfortable, sustained listening.

Unlike the text generation systems we explored in Part XXVIII, where we modeled sequences of discrete tokens, TTS operates in the continuous domain of audio waveforms. A waveform is a high-dimensional signal, typically sampled 16,000 to 48,000 times per second, where each sample is the air pressure at a moment in time. This high sampling rate creates a large dimensional gap between text and audio: a single sentence might contain fifty characters but generate hundreds of thousands of audio samples. Generating such dense, continuous outputs directly from discrete text presents unique architectural challenges that have driven the field from concatenative synthesis, which stitched together pre-recorded speech fragments, through statistical parametric methods that modeled speech production mathematically, to today's neural approaches that learn data-driven mappings.

Modern neural TTS systems typically employ a two-stage pipeline: first, an acoustic model converts text into a compressed spectral representation (a spectrogram), and second, a vocoder reconstructs the raw waveform from these spectral features. This separation mirrors the classic source-filter model of speech production, where the vocal cords give the source (pitch) and the vocal tract acts as a filter (shaping the spectrum). By decoupling linguistic content from acoustic rendering, we can train specialized models for each task, balancing the need for semantic understanding with the demands of high-fidelity signal generation. The acoustic model focuses on linguistic competence. This keeps correct pronunciation and prosody, while the vocoder specializes in signal processing quality. This keeps the output sounds crisp and natural rather than muffled or robotic.

As we discussed in Part XII: Sequence-to-Sequence, encoder-decoder architectures with attention mechanisms excel at mapping between sequences of different lengths and modalities. TTS exemplifies this challenge: a short text phrase might produce hundreds of audio frames, requiring precise alignment between phonetic content and temporal duration. The attention mechanisms we studied in Bahdanau Attention and Luong Attention find natural application here, though TTS imposes unique constraints requiring monotonic, left-to-right alignment without the reordering allowed in machine translation. In translation, a word at the end of the source sentence might appear at the beginning of the target sentence, but in speech, time flows inexorably forward, and the acoustic realization of text must follow the written order precisely.

This chapter examines the complete neural TTS pipeline, from text normalization through acoustic modeling to neural vocoding. We will explore how architectures like Tacotron use recurrent networks and attention to generate mel-spectrograms, why non-autoregressive models like FastSpeech let real-time synthesis, and how vocoders like HiFi-GAN and WaveNet reconstruct waveforms with high fidelity. Along the way, we will address the evaluation challenges specific to speech synthesis and the trade-offs between output quality and speed, including how much control the system exposes that define modern TTS systems.

A complete TTS system comprises three distinct stages: text analysis, acoustic modeling, and vocoding. Understanding the data flow through these stages clarifies why we use specific architectures at each step. Text enters as raw Unicode characters, turns into linguistic features, then into spectral representations, and finally into raw audio samples suitable for playback through speakers or headphones.

Before neural processing begins, raw text undergoes front-end normalization to resolve ambiguities that affect pronunciation. As we saw in Part I: Text Normalization, written text contains extensive context-dependent variation. Numbers require expansion into words, but the specific expansion depends on context: "1995" might be read as "nineteen ninety-five" when referring to a year, but "one thousand nine hundred ninety-five" in a mathematical context. Abbreviations present similar challenges: "Dr." expands to "Doctor" in most contexts, but might be read as "Drive" in an address. Heteronyms, words spelled identically but pronounced differently based on meaning, require semantic analysis: "read" rhymes with "reed" in present tense but with "red" in past tense, and "lead" can refer to a metal or the act of guiding.

The front-end typically performs several necessary transformations:

• Tokenization and normalization: Expanding abbreviations, formatting numbers, handling punctuation, and standardizing whitespace and special characters
• Grapheme-to-phoneme (G2P) conversion: Mapping written words to phonetic representations (ARPAbet or IPA), needed for handling out-of-vocabulary words that do not appear in the training lexicon
• Prosodic phrase prediction: Identifying boundaries where pauses or intonation changes occur, such as clause boundaries or commas, to ensure natural rhythm

While early neural TTS systems relied on lexicons and rule-based G2P systems that required extensive linguistic expertise to develop, modern end-to-end systems often learn these mappings implicitly from character or phoneme inputs. However, explicit phonetic input generally improves pronunciation accuracy for rare words, proper nouns, and technical terminology that the model might not have encountered frequently during training. The choice between character-based and phoneme-based input involves a trade-off between ease of use and accuracy: character models require no linguistic preprocessing but may mispronounce unusual words, while phoneme models require a G2P system but reach higher reliability on challenging vocabulary.

Grapheme-to-phoneme conversion itself is a non-trivial problem that benefits from the sequence-to-sequence models we studied earlier in this handbook. Neural G2P models treat pronunciation as a character-level translation task: given the spelled form of a word, predict its phoneme sequence. These models learn common English spelling-to-sound correspondences (where "tion" almost always maps to /SH AH N/) alongside the morphological and etymological patterns that govern exceptions, such as the difference between "read" in present versus past tense. For proper nouns and technical terms, neural G2P models can learn from large pronunciation lexicons, then generalize to new words by recognizing morphological patterns from known words. The quality of G2P directly limits TTS quality: mispronounced words break the listener's trust and can obscure meaning, making reliable G2P a necessary component of production systems.

Prosodic phrase prediction is another front-end stage that materially affects naturalness. Phrase boundaries determine where the synthesizer inserts pauses and resets its pitch declination, and incorrect boundaries fragment naturally coherent phrases or run together phrases that should be separated. A sentence like "The horse raced past the barn fell" requires the reader to re-parse and assign a phrase boundary after "barn" to understand the garden-path structure, but a TTS system that processes text left-to-right without syntactic awareness will likely produce a misleading reading. Advanced front-ends use syntactic parsing or neural sequence labeling to predict prosodic boundaries, incorporating part-of-speech tags, dependency relations, and sentence length as features. Modern end-to-end systems with transformer encoders can capture some of this structure implicitly, but they remain vulnerable to syntactically unusual inputs that fall outside the distribution of their training text.

The acoustic model is the core intelligence of the TTS system, learning the mapping from linguistic representations to acoustic features. Rather than predicting raw waveforms directly, which would require modeling tens of thousands of samples per second, acoustic models generate mel-spectrogram log-magnitude representations of the short-time Fourier transform (STFT) mapped to the mel scale, a perceptual frequency scale that approximates human hearing sensitivity.

Mel-spectrograms serve as an intermediate representation for several important reasons. First, they discard phase information, which is difficult to predict due to its high temporal resolution and sensitivity to small timing variations, but can be reconstructed by vocoders using either classical signal processing or learned priors. Second, they operate at a lower temporal resolution than raw audio (e.g., 80 frames per second vs. 24,000 samples per second), making sequence modeling tractable and letting the acoustic model to focus on the slowly evolving spectral envelope rather than rapid oscillations. Third, they separate the linguistic content, which is captured in the spectral envelope and formant frequencies, from the fine-grained excitation details, such as periodic glottal pulses and noise, which are handled by the vocoder. This separation aligns with the source-filter model of speech production and allows each component to specialize appropriately.

The vocoder completes the pipeline by inverting the mel-spectrogram into a time-domain waveform. This phase reconstruction problem is underdetermined: multiple waveforms can produce the same magnitude spectrogram, differing only in their phase components. Classical approaches like the Griffin-Lim algorithm iteratively estimate phase through alternating projections between the time and frequency domains, but these methods produce artifacts such as metallic buzzing and lack the naturalness of neural approaches because they rely on mathematical optimization rather than learned priors about natural speech.

Neural vocoders learn to sample from the distribution of possible waveforms conditioned on the spectrogram, using autoregressive, flow-based, GAN-based, or diffusion-based architectures. These approaches do not simply invert the spectrogram mathematically; instead, they generate waveforms that are consistent with the spectral envelope while possessing the statistical properties of natural speech, including appropriate phase relationships and temporal fine structure. We will examine these approaches in detail later in this chapter.

The dominant paradigm for acoustic modeling treats TTS as a sequence-to-sequence translation problem, mapping from a sequence of characters or phonemes to a sequence of mel-spectrogram frames. This approach, pioneered by Tacotron, uses the attention mechanisms we studied in Part XII to handle the length discrepancy between text and audio. Unlike machine translation where source and target sequences might be similar in length, TTS involves significant expansion: a single phoneme might generate dozens of acoustic frames, requiring the model to learn temporal allocation.

Tacotron 2 exemplifies the encoder-decoder approach to TTS. The architecture consists of five components working in concert:

1. Encoder: Converts input character or phoneme sequences into hidden representations that capture linguistic context
2. Attention mechanism: Aligns encoder states with decoder time steps, determining which input characters to focus on when generating each acoustic frame
3. Decoder: Autoregressively predicts mel-spectrogram frames one at a time, conditioning each prediction on previously generated frames
4. Post-net: Refines decoder outputs with residual convolutions to capture fine spectral details and reduce artifacts
5. Stop token prediction: Determines when to halt generation dynamically based on the content

The encoder processes input characters through a convolutional pre-net followed by a bidirectional LSTM. As we discussed in Part XI: Bidirectional RNNs, bidirectional processing ensures that each character's representation incorporates both left and right context, which is needed for determining pronunciation from surrounding letters. For example, the "g" in "great" requires looking ahead to the "r" to know it should be pronounced as a voiced velar stop rather than the voiced palato-alveolar affricate found in "genre."

The decoder operates autoregressively, similar to the language models in Part XXVIII. At each time step $t$, the decoder receives the mel-spectrogram frame predicted at step $t-1$ (or the ground truth during training via Teacher Forcing), processes it through a pre-net and feeds it into an autoregressive LSTM. The LSTM hidden state queries the encoder outputs through an attention mechanism to determine which input characters to focus on when generating the current acoustic frame. This autoregressive structure ensures temporal coherence in the generated spectrogram, as each frame builds upon the acoustic context established by previous frames.

Standard Bahdanau Attention computes alignment scores based solely on content matching between query and key vectors. However, TTS requires monotonic alignment: the model should attend to input characters in left-to-right order without skipping or repeating. Speech is strictly sequential; unlike machine translation where distant word pairs might align due to grammatical reordering, acoustic features must follow the phonetic sequence exactly. If the model attends to the final phoneme while still generating audio for the first phoneme, the result is unintelligible babbling.

Tacotron 2 employs location-sensitive attention, which augments content-based scoring with location information. The alignment score for decoder step $i$ attending to encoder step $j$ incorporates the previous alignment weights:

where:

• $e_{i,j}$: the alignment score (energy) for decoder step $i$ attending to encoder step $j$
• $v$: a learned weight vector for computing the attention score
• $W_q$: weight matrix for the decoder query (hidden state $h_i$)
• $h_i$: the hidden state of the decoder at step $i$
• $W_k$: weight matrix for the encoder key (hidden state $h_j$)
• $h_j$: the hidden state of the encoder at position $j$ (the value being attended to)
• $W_f$: weight matrix for the location features
• $f_{i,j}$: the convolutional features of the previous alignment vector at position $j$
• $b$: bias term

By convolving the previous attention weights with learned filters, the model can sense whether it has been attending to the left or right of the current position, encouraging forward movement through the text. This prevents the "babbling" artifacts that occur when attention gets stuck oscillating between two input positions or jumps erratically through the sequence. The convolutional features give a sense of momentum. This keeps once the model starts processing a particular phoneme, it continues to generate the appropriate number of acoustic frames before moving to the next.

The attention weights are normalized across the input sequence using a softmax, creating a context vector:

where:

• $c_i$: the context vector at decoder step $i$, representing a weighted sum of relevant encoder states
• $\alpha_{i,j}$: the attention weight showing the probability (between 0 and 1) of attending to encoder position $j$ when generating decoder output $i$
• $h_j$: the hidden state of the encoder at position $j$ (the value being attended to)
• $j$: the index iterating over all encoder positions

The attention weights are computed via softmax normalization:

where:

• $\alpha_{i,j}$: the normalized attention weight for decoder step $i$ and encoder step $j$, computed as the softmax of alignment scores
• $e_{i,j}$: the alignment score (energy) for encoder position $j$ used in the numerator
• $e_{i,k}$: the alignment score for encoder position $k$ used in the denominator summation
• $k$: the index iterating over all encoder positions for normalization (so $\sum_j \alpha_{i,j} = 1$)

The softmax ensures that the attention weights form a valid probability distribution across the input sequence, letting the model to focus on specific phonemes while generating each acoustic frame. This context vector, concatenated with the decoder LSTM output, predicts the mel-spectrogram frame through a linear projection, effectively conditioning the acoustic output on the relevant linguistic content.

Unlike text generation where we typically generate fixed-length sequences or use special end-of-sequence tokens, TTS must dynamically determine when speech concludes. Different utterances have different lengths, and the model must learn to recognize when it has generated sufficient acoustic frames to stand for the complete input text. Tacotron 2 adds a binary classification head predicting a "stop token" probability. When this probability exceeds a threshold (typically 0.5), generation terminates.

During training, the model minimizes two losses simultaneously:

where:

• $\mathcal{L}_{\text{TTS}}$: the total loss function for the TTS model, combining reconstruction and termination objectives
• $\mathcal{L}_{\text{mel}}$: the mel-spectrogram reconstruction loss (typically L1 or L2 distance), measuring how well the predicted spectrogram matches the ground truth
• $\mathcal{L}_{\text{stop}}$: the stop token prediction loss (binary cross-entropy), determining when the model should halt generation

The model optimizes both objectives simultaneously. This keeps accurate spectral prediction while learning appropriate stopping criteria for variable-length utterances. The mel loss is typically the L1 or L2 distance between predicted and ground-truth spectrograms, capturing the fine details of the spectral envelope, while the stop token loss uses binary cross-entropy. Because the large majority of frames are non-terminal (only the final frame should predict stop), the stop token loss is usually weighted to account for class imbalance. This keeps the model does not just learn to never predict termination.

While Tacotron 2 produces high-quality speech, its autoregressive nature prevents parallel generation. Each spectrogram frame must wait for the previous frame to be computed, resulting in synthesis speeds slower than real-time on standard hardware. This latency proves unacceptable for interactive applications like voice assistants, where users expect immediate responses, or for real-time captioning systems that must generate audio synchronized with live video.

Non-autoregressive TTS models, exemplified by FastSpeech and FastSpeech 2, eliminate sequential dependencies within the decoder, letting parallel generation of entire spectrograms. The key challenge is determining duration: how many acoustic frames correspond to each input phoneme? In autoregressive models, this duration emerges naturally from the attention alignment process. If the model attends to the phoneme "ae" (as in "cat") for 10 consecutive frames, that phoneme has a duration of 10 frames. Non-autoregressive models must explicitly model this duration mapping because they lack the sequential generation process that would otherwise reveal it.

In autoregressive models, the attention mechanism implicitly learns duration through the alignment process. If the model attends to the phoneme "ae" (as in "cat") for 10 consecutive frames, that phoneme has a duration of 10 frames. Non-autoregressive models must explicitly model this duration mapping because they cannot rely on the autoregressive generation process to reveal how long each phoneme should last.

FastSpeech 2 introduces a duration predictor, a small neural network trained to predict the length of each phoneme in frames. During inference, the model uses these predictions to expand the phoneme sequence through a length regulator: if phoneme $i$ has predicted duration $d_i$, it is repeated $d_i$ times along the time dimension before feeding into the spectrogram decoder. This expansion turns a sequence of linguistic features into a sequence of acoustic features with the appropriate temporal structure.

Mathematically, given encoder outputs $H = [h_1, h_2, ..., h_n]$ and durations $D = [d_1, d_2, ..., d_n]$, the length regulator constructs the expanded sequence:

where:

• $H_{\text{expanded}}$: the expanded sequence of encoder outputs with each phoneme repeated according to its predicted duration
• $h_i$: the encoder output (hidden state) for the $i$-th phoneme
• $d_i$: the predicted duration (number of acoustic frames) for the $i$-th phoneme
• $n$: the total number of phonemes in the input sequence

The duration predictor is trained using Monotonic Alignment Search (MAS) or by extracting durations from a pretrained autoregressive teacher model. FastSpeech 2 also introduces variance predictors for pitch and energy (volume), letting fine-grained control over prosody without requiring explicit linguistic features. These variance predictors let the model to generate speech with varying intonation and stress, rather than creating flat, monotone output.

Just as the Transformer Architecture changed text processing, it has been adapted for TTS with architectures like Transformer TTS and its successor, the Neural Speech Synthesis models. These replace LSTM encoders and decoders with self-attention mechanisms. This makes possible better long-range dependencies for prosody modeling across phrases. Self-attention allows the model to directly relate distant phonemes, capturing coarticulation effects where the pronunciation of one sound is influenced by sounds several positions away. When generating the pitch contour for a multi-clause sentence, the model can relate the end of the first clause to the beginning of the second. This produces the gradual pitch declination characteristic of natural declarative speech.

However, pure self-attention lacks the inductive bias toward monotonic alignment inherent in recurrent models. Recurrent networks naturally process sequences in order, while self-attention treats all positions equally and symmetrically. This symmetry is desirable in text understanding but problematic in TTS, where strict left-to-right progress through the phoneme sequence is mandatory. Consequently, Transformer TTS systems often use positional encodings more aggressively or employ Relative Position Encodings to maintain sequential structure. Some variants augment the cross-attention with a diagonal attention prior, implemented as a Gaussian that penalizes large deviations from the expected monotonic diagonal, discouraging the attention mechanism from jumping erratically through the input.

The non-autoregressive variant FastSpeech uses a Feed-Forward Transformer (FFT) block consisting of self-attention and 1D convolutional layers, processing entire sequences in parallel once durations are determined. Each FFT block applies multi-head self-attention to capture global context, followed by two position-wise convolutional layers that model local spectral patterns. Stacking several FFT blocks in both the phoneme encoder and the mel-frame decoder creates a hierarchical representation: lower layers capture individual phoneme acoustics while higher layers integrate prosodic patterns over longer spans. This architecture generates spectrograms hundreds of times faster than autoregressive models while maintaining comparable quality, making real-time on-device synthesis practical.

FastSpeech 2 extends this framework by adding variance adaptors that condition generation on predicted pitch, energy (root-mean-square amplitude), and duration simultaneously. Rather than forcing the model to infer all prosodic dimensions implicitly from the text, these explicit predictors give fine-grained controllable handles. A user can increase the pitch predictor output to raise intonation, multiply the energy predictor to emphasize certain words, or scale the duration predictor to slow down delivery for clarity. This explicit decomposition also simplifies the training signal: each predictor has a dedicated supervised objective with ground-truth values extracted from reference audio, rather than relying solely on the spectrogram reconstruction loss to teach all prosodic properties at once.

The variance adaptor for pitch typically operates on the log-basic-frequency (log-F0) contour. Rather than predicting raw F0 in Hertz, the model predicts the logarithm of F0, which turns the typically right-skewed pitch distribution into a more Gaussian-shaped distribution easier for a neural network to model. During training, forced alignment tools extract ground-truth F0 values from reference recordings using pitch detection algorithms. At inference, the model predicts a normalized pitch shift per phoneme that is added to a speaker-specific mean pitch. This allows the system to transfer prosodic patterns from the training distribution while respecting the speaker's baseline range.

The mel-spectrogram gives a compact, perceptually relevant representation, but it discards phase information. Reconstructing a waveform requires estimating these missing phases, a problem classical signal processing solves through iterative algorithms that often produce metallic artifacts or temporal smearing. Neural vocoders learn to generate waveforms directly, conditioned on spectrograms, creating materially more natural speech with crisp transients and appropriate noise characteristics.

WaveNet, developed by DeepMind, pioneered autoregressive neural vocoding. Operating directly on the raw audio waveform, WaveNet models the joint probability of audio samples as a product of conditional probabilities:

where:

• $p(x)$: the joint probability of the audio waveform $x$ consisting of $T$ samples
• $T$: the total number of audio samples in the waveform
• $t$: the current time step index (from 1 to $T$)
• $x_t$: the audio sample at time step $t$
• $x_1, ..., x_{t-1}$: all previous audio samples before time $t$
• $c$: the conditioning information (the mel-spectrogram)
• $p(x_t | x_1, ..., x_{t-1}, c)$: the conditional probability of sample $x_t$ given previous samples and conditioning

This mirrors the autoregressive language modeling we discussed in Part XXII: Causal Language Modeling, but applied to continuous audio samples rather than discrete tokens. The model learns the statistical dependencies between consecutive samples. This keeps the generated waveform is locally consistent and free of discontinuities.

WaveNet uses dilated causal convolutions, which are convolutions with gaps (dilations) that exponentially increase the receptive field while maintaining causality (no access to future samples). A stack of dilated convolutions with dilation rates $1, 2, 4, 8, ..., 512$ allows the model to capture temporal dependencies across thousands of samples (hundreds of milliseconds) using relatively few layers. The causal constraint ensures that the prediction for sample $t$ depends only on samples $1$ through $t-1$, preserving the autoregressive property and preventing the model from "cheating" by looking at future values during training.

The model outputs a categorical distribution over quantized audio amplitudes (typically 8-bit or 16-bit $\mu$-law encoded), sampled autoregressively to generate waveforms. While WaveNet produces excellent quality with rich timbre and natural-sounding speech, its sample-by-sample generation is computationally prohibitive for real-time applications, often requiring specialized hardware like GPUs or TPUs to reach acceptable speeds.

WaveRNN addresses this by using a single-layer recurrent network with dual softmax outputs for coarse and fine quantization levels, reducing computational complexity while maintaining quality. The dual softmax splits the prediction of 16-bit audio into two 8-bit predictions (coarse and fine), simplifying the probability distribution. However, both models remain slower than real-time without specialized hardware acceleration, limiting their deployment in resource-constrained environments.

Generative Adversarial Networks give an alternative path to fast, high-quality vocoding. Rather than modeling the exact likelihood of each audio sample, which requires expensive autoregressive computation, GAN vocoders learn to produce waveforms that are indistinguishable from real speech according to a discriminator network. This adversarial approach trades exact probabilistic modeling for speed and perceptual quality.

MelGAN and HiFi-GAN stand for the state of the art in efficient neural vocoding. These models use transposed convolutions to upsample mel-spectrograms to audio rates, followed by residual blocks with dilated convolutions to model temporal dependencies. The generator operates non-autoregressively, creating audio frames in parallel, which lets real-time synthesis on standard hardware.

Training employs multiple discriminators operating at different time scales:

• A multi-scale discriminator examines the waveform at different resolutions (raw audio, average-pooled by 2, by 4), capturing structure at various temporal granularities
• A multi-period discriminator examines different periodic patterns (every 2nd sample, every 3rd sample, etc.), capturing pitch periodicity and harmonic structure that are important for natural-sounding speech

The adversarial loss encourages the generator to match the distribution of natural speech:

where:

• $\mathcal{L}_{\text{adv}}$: the adversarial loss function training the generator to fool the discriminator
• $\mathbb{E}_{(x,s)}$: the expectation over real audio $x$ and corresponding spectrogram $s$ pairs from the training data
• $x$: real audio waveform sampled from the data distribution
• $s$: the input mel-spectrogram used as conditioning for generation
• $D$: the discriminator network that classifies audio as real (1) or fake (0)
• $G$: the generator network (vocoder) that synthesizes audio from spectrograms
• $\mathbb{E}_s$: the expectation over generated samples produced from spectrogram $s$

Additional feature matching losses (L1 distance between intermediate discriminator layer activations for real vs. generated audio) and mel-spectrogram reconstruction losses stabilize training and ensure that the generated audio remains faithful to the conditioning spectrogram, preventing mode collapse where the generator produces plausible-sounding audio that does not match the input text.

HiFi-GAN reaches real-time synthesis on CPUs while matching WaveNet quality by using a multi-receptive field fusion (MRF) module that processes audio through parallel dilated convolution branches with different dilation rates, similar to the Inception architecture in computer vision. This allows the model to capture both short-term transients (like consonant bursts) and long-term structure (like pitch periodicity) simultaneously.

Flow-based models like WaveGlow and WaveFlow give an alternative to adversarial training, learning an invertible transformation between a simple noise distribution and the complex distribution of audio waveforms. These models allow exact likelihood computation and efficient sampling through the change of variables formula. Given an invertible function $f$ that turns a latent noise vector $z$ into an audio sample $x = f(z)$, the log-likelihood of $x$ is:

where:

• $\log p(x)$: the log-likelihood of the audio waveform $x$ under the model
• $\log p(z)$: the log-likelihood of the latent variable $z$ under the base distribution (typically a standard Gaussian)
• $f$: the learned invertible transformation mapping from latent space $z$ to data space $x$
• $z$: the latent noise variable drawn from a simple base distribution
• $\frac{\partial f}{\partial z}$: the Jacobian matrix of partial derivatives of $f$ with respect to $z$
• $\det \frac{\partial f}{\partial z}$: the determinant of the Jacobian, accounting for the volume change under the transformation

The Jacobian determinant term is what makes normalizing flows computationally tractable. For a general invertible function, computing this determinant requires $O(d^3)$ operations for $d$-dimensional data, which makes it prohibitive for raw audio. Flow models avoid this by constructing $f$ as a composition of simple bijections with triangular or block-diagonal Jacobians, for which the determinant reduces to a product of diagonal elements. WaveGlow uses affine coupling layers where each layer splits the input into two halves, uses one half to predict scale and shift parameters applied to the other half, and the resulting Jacobian is triangular by construction.

Flow-based vocoders train stably and allow exact likelihood evaluation, which is useful for model selection and debugging. They avoid the training instabilities associated with GAN discriminators and the exposure bias of autoregressive models. However, they historically lagged GANs in perceptual quality, though recent architectures have narrowed this gap through improved coupling layers and conditioning mechanisms that incorporate the mel-spectrogram more deeply into each transformation stage.

Diffusion-based vocoders stand for the most recent advancement in this space. Models like DiffWave and WaveGrad treat waveform generation as a denoising problem: starting from Gaussian noise, the model iteratively refines the signal over $T$ diffusion steps, each step guided by the mel-spectrogram conditioning. At each step $t$, the model predicts the noise component $\epsilon$ that was added to the clean signal at that diffusion level, and subtracts an estimate of this noise to take one step toward the clean waveform. Because the model learns a smooth denoising function rather than a single direct mapping, diffusion models can capture complex multimodal distributions in the audio space, creating outputs with rich textural detail. The primary limitation is inference speed: achieving high quality typically requires 50 to 200 diffusion steps, each of which involves a full neural network forward pass. Recent work has focused on distilling diffusion models into fewer steps, using progressive distillation techniques that teach a student model to jump over multiple original steps in a single stride, achieving real-time performance with as few as 4 to 6 inference steps.

The two-stage paradigm, with a separate acoustic model creating spectrograms and a separate vocoder reconstructing waveforms, dominated neural TTS for several years. While effective, this separation introduces a training mismatch: the acoustic model is trained to minimize spectrogram reconstruction error, but the vocoder is trained to produce natural-sounding audio from spectrograms. Any imperfections in the predicted spectrogram, such as slightly blurred formants or missing fine detail in fricative regions, propagate into the vocoder's input at inference time even though the vocoder was never trained to handle such imperfect inputs. This train-test mismatch can cause the vocoder to amplify subtle acoustic model errors into audible artifacts.

End-to-end TTS models sidestep this mismatch by training a single model that maps text directly to waveforms, optimizing a perceptual loss that shows the final audio quality rather than an intermediate spectrogram distance. The most influential end-to-end approach is VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech), which combines a conditional variational autoencoder with adversarial training to produce a model capable of generating high-quality speech from phoneme sequences in a single forward pass.

VITS frames TTS as a conditional variational autoencoder (CVAE) problem. The encoder, called the posterior encoder, takes as input the raw audio waveform (specifically, its linear-scale spectrogram) during training and encodes it into a latent variable $z$ that captures the stochastic aspects of the audio. A normalizing flow, called the prior flow, turns a simple Gaussian conditioned on the text encoder output into a distribution that approximates the posterior. At inference time, the prior flow samples $z$ from the text-conditioned prior, and a HiFi-GAN decoder renders this latent into a waveform. The training objective combines three terms:

1. The reconstruction loss: how well the decoded audio matches the original recording, measured in the mel-spectrogram domain
2. The KL divergence: how closely the prior distribution matches the posterior distribution, enforcing that the text encoder learns to predict what the audio encoder observes
3. The adversarial loss: how convincingly the generated audio fools a multi-period discriminator identical to the one used in HiFi-GAN

The CVAE framework creates a principled probabilistic foundation for the training signal. The KL divergence term forces the text encoder to learn a rich intermediate representation that captures the linguistic content and the prosodic and speaker-specific aspects necessary to reconstruct the waveform. Because both encoder and decoder are trained jointly on real audio, the model learns to generate waveforms that are consistent with natural speech statistics without the train-test mismatch of the two-stage pipeline.

VITS also introduces stochastic duration prediction using a flow-based duration model. Unlike the deterministic duration predictor in FastSpeech 2, which outputs a single duration estimate per phoneme, VITS models duration as a latent variable with a learned prior and posterior. This allows the model to sample different duration values at inference time, creating natural prosodic variation across multiple readings of the same text, rather than always generating identically timed speech. The stochastic nature mirrors how human speakers naturally vary their timing with each utterance while maintaining overall intelligibility.

The practical result is that VITS reaches near-human quality in MOS evaluations while operating as a single unified model that requires no separate vocoder fine-tuning or inference-time alignment. The architecture generalizes naturally to multi-speaker and cross-lingual scenarios by conditioning on speaker embeddings or language tokens, which makes it a versatile foundation for production TTS systems. Facebook's MMS (Massively Multilingual Speech) TTS model, which we use in the code implementation below, builds directly on the VITS framework to support over 1,000 languages with a single model architecture.

Evaluating speech synthesis presents unique challenges. Unlike text generation where we can compare against reference strings using BLEU or ROUGE (covered in Part LIII), audio quality involves perceptual dimensions that resist simple mathematical characterization. Naturalness depends on subtle cues like breathiness, vocal effort, and prosodic appropriateness that are difficult to quantify with simple distance metrics.

The gold standard for TTS evaluation remains human judgment, typically quantified through Mean Opinion Score (MOS) studies. In these studies, listeners rate synthesized speech on a 5-point scale:

1. Bad (unintelligible, unnatural)
2. Poor (intelligible but annoying)
3. Fair (slightly annoying)
4. Good (perceptible distortion, not annoying)
5. Excellent (imperceptible distortion)

CMOS (Comparative Mean Opinion Score) presents listeners with paired samples (A and B) and asks which is better. This gives relative quality assessments useful for comparing system variants without requiring absolute calibration.

While MOS gives the most reliable quality assessment, it is expensive, time-consuming, and suffers from variability across listener populations and testing conditions. Different demographic groups may have different expectations for synthetic speech, and laboratory conditions may not reflect real-world usage scenarios. Despite these limitations, MOS remains needed for final quality validation, particularly for commercial systems.

Objective metrics attempt to approximate human perception algorithmically, giving faster and cheaper evaluation during development:

Mel Cepstral Distortion (MCD) measures the Euclidean distance between mel-frequency cepstral coefficients (MFCCs) of synthesized and reference audio. MFCCs capture the spectral envelope in a compact representation inspired by human auditory processing. Lower MCD indicates better spectral envelope matching:

where:

• $\text{MCD}$: the Mel Cepstral Distortion measuring spectral difference between synthesized and reference audio
• $D$: the total number of cepstral coefficients (dimensions) used in the comparison
• $d$: the index iterating over cepstral coefficients from $1$ to $D$
• $c_d$: the $d$-th MFCC coefficient of the reference (ground truth) audio
• $\hat{c}_d$: the $d$-th MFCC coefficient of the synthesized audio
• $\frac{10}{\ln 10}$: a scaling constant (approximately 4.3429) converting natural logarithm units to decibels

However, MCD correlates poorly with human perception of naturalness as it penalizes all spectral deviations equally regardless of perceptual salience. Some spectral differences are inaudible to humans, while others are highly noticeable, but MCD treats them identically.

PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility) predict subjective quality and intelligibility respectively. PESQ models human auditory perception by comparing internal representations of reference and degraded speech, accounting for frequency masking and loudness perception. STOI focuses specifically on intelligibility, correlating highly with word recognition rates in noisy conditions, which makes it useful for applications where clarity is the primary requirement.

Fundamental Frequency (F0) metrics assess prosody quality by comparing pitch contours between synthesized and reference speech. Metrics like F0 Frame Error (FFE) measure the percentage of frames where the predicted pitch differs materially from the reference, helping to quantify whether the model has learned appropriate intonation patterns.

As we discussed in Embedding Evaluation, TTS evaluation can be intrinsic (measuring acoustic quality directly) or extrinsic (measuring performance on downstream tasks). Word Error Rate (WER) from speech recognition systems gives an extrinsic metric: if a speech recognizer transcribes synthesized speech accurately, the TTS system likely produces intelligible, natural-sounding output. This "TTS-for-ASR" evaluation proves particularly useful for evaluating low-resource languages where human MOS studies are impractical or for rapidly iterating on system designs without organizing listening tests.

Let's implement a complete neural TTS pipeline using modern libraries. We will use a pre-trained model from the transformers library to generate speech, then examine the intermediate representations to understand the acoustic modeling and vocoding stages.

The pitch contour reveals the prosodic patterns of the synthesized utterance. Natural speech exhibits continuous variation in basic frequency (F0) to convey emphasis and emotion as well as syntactic structure. The smoothness of this contour indicates that the model has learned appropriate prosody rather than generating robotic, monotone speech.

The VITS TTS system demonstrated above exposes several important configuration choices, each with real trade-offs between output quality and speed under a given resource budget. Understanding these parameters helps you adapt the pipeline to specific applications.

The sampling rate (16,000 Hz in the MMS model) determines the temporal density of the audio signal and thus the maximum representable frequency, limited by the Nyquist theorem to half the sampling rate (8,000 Hz at 16kHz). Telephone-quality audio typically operates at 8,000 Hz, which captures sufficient bandwidth for intelligible speech but misses the presence frequencies (around 4,000 to 8,000 Hz) that contribute to the clarity of fricatives like "s" and "f." High-fidelity applications such as audiobook narration or professional voice assistants use 22,050 Hz or 44,100 Hz to preserve the full audible range. Higher sampling rates increase both the length of the waveform sequence and the computational load of the vocoder, since the decoder must produce more samples per second. For the VITS architecture, this trade-off is particularly acute because the HiFi-GAN decoder processes all samples in parallel, so doubling the sampling rate roughly doubles the memory footprint.

The number of mel bins (80 in this implementation, with 128 becoming common in newer systems) controls the spectral resolution of the intermediate representation. Each bin corresponds to a frequency band on the mel scale, and 80 bins give sufficient resolution to distinguish different vowel qualities and consonant types in English. Increasing to 128 bins captures finer spectral detail, which can improve the quality of sibilants and fricatives, but widens the target for the acoustic model and requires proportionally more model capacity to generate accurately. Models trained for tonal languages like Mandarin sometimes use more bins to resolve the subtle spectral differences in tone contrasts.

The hop length and FFT window size together define the time-frequency trade-off in the spectrogram. A hop length of 256 samples at 16,000 Hz yields approximately 80 frames per second (frame shift of 12.5ms), which is short enough to track rapidly changing sounds like stop consonants but long enough to keep the sequence manageable for the acoustic model. The FFT window of 1,024 samples (64ms) gives good frequency resolution at the cost of temporal smearing: brief events shorter than the window duration blend across adjacent frames. In practice, these values are chosen to match the acoustic properties of speech, where phonemic transitions occur on the order of 20 to 50ms and spectral resolution below 100 Hz matters for distinguishing formant patterns.

The vocabulary size depends entirely on the tokenization scheme. Character-based models have small vocabularies (perhaps 100 to 200 entries for the characters and special tokens of a given language) and require no external G2P system, but must learn pronunciation from distributional context. Phoneme-based models typically use 40 to 80 phoneme symbols per language and produce more reliable pronunciation of rare words, at the cost of requiring a G2P preprocessing step. The MMS model uses a phoneme vocabulary derived from the IPA, which generalizes across the 1,000+ supported languages because IPA was designed to stand for all human speech sounds in a unified symbol set.

These parameters interact in ways that are not always obvious. Increasing the sampling rate while keeping the hop length fixed in samples shortens the frame duration and increases the number of frames per utterance, making the acoustic modeling sequence longer and potentially harder to learn. A common solution is to scale the hop length proportionally with the sampling rate, maintaining constant frame duration regardless of audio quality level. Similarly, widening the mel filter bank beyond 80 bins often requires retraining the vocoder, since the GAN discriminators learn to detect artifacts at a resolution specific to the training configuration. When adapting a pre-trained system to a new application, understanding which parameters are tightly coupled helps identify the minimal changes needed to reach the desired quality and efficiency profile.

Despite remarkable advances, neural TTS systems face significant challenges that define active research frontiers.

Data Efficiency and Speaker Adaptation: High-quality TTS typically requires 10 to 30 hours of single-speaker recordings to capture the full range of phonetic contexts and prosodic variations. This data requirement creates a barrier for underrepresented languages, regional dialects, and individuals who want to create personalized voices. Adapting to new speakers with limited data, sometimes called few-shot speaker adaptation, remains difficult because voices differ in subtle acoustic properties that are hard to disentangle from linguistic content. A speaker's voice is characterized by pitch, speaking rate, the detailed spectral shaping of the vocal tract, the breathiness of phonation, the timing of coarticulation, and the rhythmic patterns of their particular dialect. Capturing all of these dimensions from just a few minutes of audio requires learning a rich prior over voice characteristics during multi-speaker pretraining and then efficiently updating only the speaker-specific aspects at adaptation time.

While techniques like speaker embeddings and neural speaker codes let multi-speaker modeling by conditioning generation on a vector extracted from reference audio, the mapping from a brief reference clip to the full voice timbre remains approximate. Models trained on many speakers can learn a real speaker space, but they tend to regress toward the mean of that space when reference audio is scarce, losing the most distinctive aspects of the target voice. Techniques like meta-learning and voice conversion can help, but capturing the nuances of a specific voice with only tens of seconds of audio continues to challenge current architectures.

Prosody and Emotional Controllability: Current models learn average prosodic patterns from training data but offer limited explicit control over intonation, stress, and emotional expression. The relationship between text and prosody is heavily context-dependent and involves layers of linguistic, social, and pragmatic meaning that models must infer from text alone. A single sentence like "I didn't say he stole the money" carries seven different meanings depending on which word receives primary stress, yet a standard TTS system will produce the same intonation pattern each time regardless of the intended emphasis. While variance predictors in FastSpeech 2 allow pitch and energy adjustment, these adjustments operate at the phoneme level and require the user to specify the desired contour explicitly, which is impractical in most applications. Fine-grained expressive control, where the user specifies an emotion or speaking style and the model renders it appropriately across the entire utterance, requires either reference-based style transfer (where a sample of the desired speaking style guides generation) or more advanced linguistic representations that encode pragmatic intent.

Real-Time Constraints: Although non-autoregressive acoustic models and GAN vocoders reach real-time synthesis on server hardware, latency remains necessary for conversational applications running on edge devices. Full pipeline latency, including text normalization, G2P conversion, acoustic modeling, and vocoding, must fall below 200 to 300 milliseconds for natural turn-taking in dialogue, because latency beyond this threshold disrupts conversational rhythm and makes interactions feel sluggish. On-device synthesis for mobile assistants faces additional memory constraints: a full TTS system can require hundreds of megabytes of model weights, competing with other applications for limited RAM. Compression techniques such as knowledge distillation, weight pruning, and quantization reduce model size and inference time but typically trade off some quality. Streaming TTS, which begins generating audio before the complete text is available, is needed for live captioning, simultaneous translation, and reading assistants, and requires architectures that can produce coherent audio from incomplete sentence prefixes without making prosodic commitments they cannot later honor when the sentence ending is revealed.

Robustness and Error Handling: Attention-based TTS systems occasionally suffer from alignment failures: skipping over words, repeating phrases multiple times, or creating unintelligible babbling for out-of-distribution inputs such as unusual proper nouns, technical terminology, or code-heavy text containing identifiers and symbols not seen during training. These failures are difficult to predict in advance and can be embarrassing or harmful in production deployments, particularly for accessibility applications where TTS is the primary means of accessing written content. The monotonic alignment constraints in location-sensitive attention reduce but do not eliminate these failures. More reliable approaches include explicit duration models that prevent temporal misalignment by construction (as in FastSpeech), guided attention losses that penalize non-diagonal attention patterns during training, and confidence-based stopping criteria that detect when the model has left its training distribution and fall back to safer synthesis strategies.

Ethical and Security Implications: High-fidelity neural TTS has made voice cloning accessible at very large scale, raising serious concerns about deepfake audio used for impersonation or fraud. A malicious actor can now clone a target speaker's voice from a brief public recording and use the resulting model to generate fraudulent audio with arbitrary content. As model quality approaches human parity, detecting synthetic speech through perceptual means becomes increasingly difficult, requiring dedicated detection systems trained to identify subtle neural artifacts. The technical arms race between synthesis and detection is ongoing, and researchers have proposed countermeasures including inaudible watermarks embedded during synthesis, provenance metadata for certified real audio, and detection models that classify audio as synthetic using statistical signatures invisible to human listeners.

The societal implications extend beyond deliberate misuse. TTS systems trained on biased datasets may underrepresent certain accents, dialects, or speaking styles, creating speech that sounds unnatural or carries unintended social connotations for speakers from underrepresented linguistic communities. A system trained primarily on broadcast English may render accented inputs with degraded naturalness or incorrect prosody, effectively giving inferior service to non-native English speakers. Ensuring equitable quality across linguistic and demographic groups requires intentional dataset curation, evaluation with diverse listener panels, and ongoing monitoring in production.

The field continues to evolve toward fully end-to-end architectures that learn text normalization and G2P mappings directly from text-audio pairs without requiring explicit linguistic preprocessing, neural audio codecs that compress audio more efficiently than mel-spectrograms using learned discrete representations such as those in EnCodec and SoundStream, and multimodal systems that generate speech synchronized with facial animation or gestures for virtual avatars and embodied conversational agents. Neural audio codecs are particularly promising because they create a discrete token sequence from audio that a language model can directly generate, potentially unifying speech synthesis with the transformer-based generation we explored throughout this handbook. As we will explore in upcoming chapters on evaluation, developing reliable metrics that correlate with human perception remains needed for advancing these systems in ways that are both technically excellent and socially responsible.

Text-to-Speech synthesis turns discrete text into continuous audio through a pipeline of neural components, each addressing specific representational challenges. The acoustic model maps normalized text or phonemes to mel-spectrograms using sequence-to-sequence architectures with monotonic attention, learning to allocate temporal duration to linguistic units while preserving spectral accuracy. The vocoder reconstructs high-fidelity waveforms from these spectral representations, solving the ill-posed phase reconstruction problem through learned generative models.

Key architectural developments have shaped the field: Tacotron 2 established the paradigm of attention-based acoustic modeling with location-sensitive alignment. This shows that neural networks could learn the complex mapping from text to spectrograms without extensive linguistic feature engineering; FastSpeech demonstrated that non-autoregressive generation with explicit duration prediction lets real-time synthesis by eliminating sequential dependencies; and HiFi-GAN showed that adversarial training produces vocoders matching autoregressive quality at orders of magnitude faster speeds. This makes possible deployment on consumer hardware.

The evaluation of TTS systems requires both subjective human judgments (MOS, CMOS) to capture perceptual naturalness and objective metrics (MCD, PESQ, STOI) to let rapid iteration during development, with the recognition that perceptual quality involves dimensions not fully captured by mathematical distance measures. As TTS quality approaches human parity, research focuses on controllability, efficiency, and the ethical deployment of voice cloning capabilities. This keeps these powerful technologies benefit society while mitigating risks of misuse.

The techniques explored here, including attention mechanisms for alignment, variance predictors for prosody, and adversarial training for high-fidelity generation, show how deep learning architectures adapt to the unique demands of audio synthesis. These same principles extend to voice conversion, speech enhancement, and neural audio generation, narrowing the boundary between synthesized and recorded sound.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about neural text-to-speech synthesis.