Text-to-Speech › Text-to-Speech

Text-to-speech systems convert written text into spoken audio, creating a direct interface between written language and acoustic signals. TTS bridges the gap between the symbolic world of text and the acoustic world of sound waves, letting applications from voice assistants and audiobook generation to accessibility tools and language learning platforms. The challenge lies in converting discrete, abstract symbols into continuous physical vibrations that carry lexical meaning and prosody, including emotion and speaker identity. A reading assistant that speaks in a flat, robotic monotone fails users even when every word is correctly pronounced, because naturalness and expressiveness are prerequisites for comfortable, sustained listening.

Unlike the text generation systems we explored in Part XXVIII, where we modeled sequences of discrete tokens, TTS operates in the continuous domain of audio waveforms. A waveform is a high-dimensional signal, typically sampled 16,000 to 48,000 times per second, where each sample is the air pressure at a moment in time. This high sampling rate creates a large dimensional gap between text and audio: a single sentence might contain fifty characters but generate hundreds of thousands of audio samples. Generating such dense, continuous outputs directly from discrete text presents unique architectural challenges that have driven the field from concatenative synthesis, which stitched together pre-recorded speech fragments, through statistical parametric methods that modeled speech production mathematically, to today's neural approaches that learn data-driven mappings.

Modern neural TTS systems typically employ a two-stage pipeline: first, an acoustic model converts text into a compressed spectral representation (a spectrogram), and second, a vocoder reconstructs the raw waveform from these spectral features. This separation mirrors the classic source-filter model of speech production, where the vocal cords give the source (pitch) and the vocal tract acts as a filter (shaping the spectrum). By decoupling linguistic content from acoustic rendering, we can train specialized models for each task, balancing the need for semantic understanding with the demands of high-fidelity signal generation. The acoustic model focuses on linguistic competence. This keeps correct pronunciation and prosody, while the vocoder specializes in signal processing quality. This keeps the output sounds crisp and natural rather than muffled or robotic.

As we discussed in Part XII: Sequence-to-Sequence, encoder-decoder architectures with attention mechanisms excel at mapping between sequences of different lengths and modalities. TTS exemplifies this challenge: a short text phrase might produce hundreds of audio frames, requiring precise alignment between phonetic content and temporal duration. The attention mechanisms we studied in Bahdanau Attention and Luong Attention find natural application here, though TTS imposes unique constraints requiring monotonic, left-to-right alignment without the reordering allowed in machine translation. In translation, a word at the end of the source sentence might appear at the beginning of the target sentence, but in speech, time flows inexorably forward, and the acoustic realization of text must follow the written order precisely.

This chapter examines the complete neural TTS pipeline, from text normalization through acoustic modeling to neural vocoding. We will explore how architectures like Tacotron use recurrent networks and attention to generate mel-spectrograms, why non-autoregressive models like FastSpeech let real-time synthesis, and how vocoders like HiFi-GAN and WaveNet reconstruct waveforms with high fidelity. Along the way, we will address the evaluation challenges specific to speech synthesis and the trade-offs between output quality and speed, including how much control the system exposes that define modern TTS systems.