Speech Representations › Speech Representations: Mel Spectrograms

Speech exists as a continuous pressure wave traveling through air, a fundamentally different modality from the discrete symbols of text we have processed throughout this book. Unlike the clean, segmented tokens of written language, speech is a dense, high-dimensional signal where meaning is encoded in subtle variations of air pressure over time. Before transformers can process spoken language, we must bridge the gap between the analog acoustic world and the digital feature spaces where neural networks operate. This transformation involves capturing the spectral characteristics of sound while respecting the constraints of human hearing and the computational requirements of machine learning.

The raw waveform, a high-resolution sequence of air pressure measurements, contains all the information in speech but presents it in a form that is computationally unwieldy and perceptually redundant. A single second of CD-quality audio contains 44,100 samples, creating sequences thousands of times longer than typical text inputs. Feed this directly into a transformer, and the quadratic cost of self-attention becomes prohibitive: processing just one second would require computing attention over 44,100 positions, resulting in nearly two billion pairwise comparisons. More importantly, raw samples represent temporal evolution but obscure the frequency content that distinguishes phonemes. While the waveform shows how pressure changes from moment to moment, it does not directly reveal which frequencies are present, which makes it difficult for models to identify the vowel formants or consonant bursts that characterize speech sounds.

The spectral envelope, the distribution of energy across frequencies, determines whether we hear an "ah" or an "ee," while the fine temporal structure carries speaker-specific nuances and pitch information. These two aspects of speech are intertwined in the raw waveform but separable in the frequency domain, which is why almost every successful speech recognition system begins by transforming to a spectral representation.

This chapter develops the representations that make speech tractable for neural networks. We move from time-domain waveforms to frequency-domain spectrograms, then compress these into perceptually motivated mel spectrograms that align with human hearing. We examine the preprocessing pipeline that prepares these features for model training, including normalization techniques that handle the variability inherent in acoustic environments, recording equipment, and speaker characteristics. These foundations prepare us for the speech recognition architectures we will explore in subsequent chapters, where these representations serve as the input to encoder-decoder models like Whisper. Understanding this preprocessing pipeline is needed for implementing speech systems, diagnosing failures, optimizing for specific acoustic conditions, and adapting models to new languages or recording environments.