Text-to-Speech › The Neural TTS Pipeline

A complete TTS system comprises three distinct stages: text analysis, acoustic modeling, and vocoding. Understanding the data flow through these stages clarifies why we use specific architectures at each step. Text enters as raw Unicode characters, turns into linguistic features, then into spectral representations, and finally into raw audio samples suitable for playback through speakers or headphones.

Before neural processing begins, raw text undergoes front-end normalization to resolve ambiguities that affect pronunciation. As we saw in Part I: Text Normalization, written text contains extensive context-dependent variation. Numbers require expansion into words, but the specific expansion depends on context: "1995" might be read as "nineteen ninety-five" when referring to a year, but "one thousand nine hundred ninety-five" in a mathematical context. Abbreviations present similar challenges: "Dr." expands to "Doctor" in most contexts, but might be read as "Drive" in an address. Heteronyms, words spelled identically but pronounced differently based on meaning, require semantic analysis: "read" rhymes with "reed" in present tense but with "red" in past tense, and "lead" can refer to a metal or the act of guiding.

The front-end typically performs several necessary transformations:

• Tokenization and normalization: Expanding abbreviations, formatting numbers, handling punctuation, and standardizing whitespace and special characters
• Grapheme-to-phoneme (G2P) conversion: Mapping written words to phonetic representations (ARPAbet or IPA), needed for handling out-of-vocabulary words that do not appear in the training lexicon
• Prosodic phrase prediction: Identifying boundaries where pauses or intonation changes occur, such as clause boundaries or commas, to ensure natural rhythm

While early neural TTS systems relied on lexicons and rule-based G2P systems that required extensive linguistic expertise to develop, modern end-to-end systems often learn these mappings implicitly from character or phoneme inputs. However, explicit phonetic input generally improves pronunciation accuracy for rare words, proper nouns, and technical terminology that the model might not have encountered frequently during training. The choice between character-based and phoneme-based input involves a trade-off between ease of use and accuracy: character models require no linguistic preprocessing but may mispronounce unusual words, while phoneme models require a G2P system but reach higher reliability on challenging vocabulary.

Grapheme-to-phoneme conversion itself is a non-trivial problem that benefits from the sequence-to-sequence models we studied earlier in this handbook. Neural G2P models treat pronunciation as a character-level translation task: given the spelled form of a word, predict its phoneme sequence. These models learn common English spelling-to-sound correspondences (where "tion" almost always maps to /SH AH N/) alongside the morphological and etymological patterns that govern exceptions, such as the difference between "read" in present versus past tense. For proper nouns and technical terms, neural G2P models can learn from large pronunciation lexicons, then generalize to new words by recognizing morphological patterns from known words. The quality of G2P directly limits TTS quality: mispronounced words break the listener's trust and can obscure meaning, making reliable G2P a necessary component of production systems.

Prosodic phrase prediction is another front-end stage that materially affects naturalness. Phrase boundaries determine where the synthesizer inserts pauses and resets its pitch declination, and incorrect boundaries fragment naturally coherent phrases or run together phrases that should be separated. A sentence like "The horse raced past the barn fell" requires the reader to re-parse and assign a phrase boundary after "barn" to understand the garden-path structure, but a TTS system that processes text left-to-right without syntactic awareness will likely produce a misleading reading. Advanced front-ends use syntactic parsing or neural sequence labeling to predict prosodic boundaries, incorporating part-of-speech tags, dependency relations, and sentence length as features. Modern end-to-end systems with transformer encoders can capture some of this structure implicitly, but they remain vulnerable to syntactically unusual inputs that fall outside the distribution of their training text.

The acoustic model is the core intelligence of the TTS system, learning the mapping from linguistic representations to acoustic features. Rather than predicting raw waveforms directly, which would require modeling tens of thousands of samples per second, acoustic models generate mel-spectrogram log-magnitude representations of the short-time Fourier transform (STFT) mapped to the mel scale, a perceptual frequency scale that approximates human hearing sensitivity.

Mel-spectrograms serve as an intermediate representation for several important reasons. First, they discard phase information, which is difficult to predict due to its high temporal resolution and sensitivity to small timing variations, but can be reconstructed by vocoders using either classical signal processing or learned priors. Second, they operate at a lower temporal resolution than raw audio (e.g., 80 frames per second vs. 24,000 samples per second), making sequence modeling tractable and letting the acoustic model to focus on the slowly evolving spectral envelope rather than rapid oscillations. Third, they separate the linguistic content, which is captured in the spectral envelope and formant frequencies, from the fine-grained excitation details, such as periodic glottal pulses and noise, which are handled by the vocoder. This separation aligns with the source-filter model of speech production and allows each component to specialize appropriately.

The vocoder completes the pipeline by inverting the mel-spectrogram into a time-domain waveform. This phase reconstruction problem is underdetermined: multiple waveforms can produce the same magnitude spectrogram, differing only in their phase components. Classical approaches like the Griffin-Lim algorithm iteratively estimate phase through alternating projections between the time and frequency domains, but these methods produce artifacts such as metallic buzzing and lack the naturalness of neural approaches because they rely on mathematical optimization rather than learned priors about natural speech.

Neural vocoders learn to sample from the distribution of possible waveforms conditioned on the spectrogram, using autoregressive, flow-based, GAN-based, or diffusion-based architectures. These approaches do not simply invert the spectrogram mathematically; instead, they generate waveforms that are consistent with the spectral envelope while possessing the statistical properties of natural speech, including appropriate phase relationships and temporal fine structure. We will examine these approaches in detail later in this chapter.