Speech Representations › Summary

Speech representations bridge the continuous acoustic world and discrete neural processing. The journey from air pressure waves to model input involves several key transformations that progressively extract and compress phonetically relevant information:

• Sampling converts analog signals to digital at rates satisfying the Nyquist criterion (typically 16 kHz for speech), preserving all frequency content up to 8 kHz without aliasing.
• Pre-emphasis compensates for spectral tilt and boosts high-frequency content. This keeps consonant information is not overshadowed by dominant low-frequency vowel energy.
• Framing and windowing segment the signal into quasi-stationary chunks (25ms frames, 10ms shift) tapered to prevent spectral leakage, respecting the time-varying nature of speech while letting spectral analysis.
• STFT turns time-domain frames into frequency-domain representations, revealing the spectral envelope that carries phonetic identity.
• Mel filterbanks aggregate linear frequency bins into perceptually spaced channels (typically 80 filters) following the logarithmic sensitivity of human hearing, allocating computational resources where they matter most for perception.
• Log compression matches the non-linear perception of loudness and stabilizes numerical ranges, preventing high-energy regions from dominating the learning process.
• Normalization (CMVN) removes channel and speaker variability, standardizing features to zero mean and unit variance so the model focuses on linguistic content rather than recording conditions.

The resulting mel spectrogram compresses a 16,000-dimensional second of audio into a $100 \times 80$ matrix (for 10ms frames and 80 mel bins), preserving phonetically relevant information while letting efficient processing by transformer encoders. This represents a 50-fold reduction in data rate from raw samples, yet retains the needed cues for phonetic discrimination. The representation balances the temporal resolution needed for consonant discrimination with the frequency resolution needed for vowel identification, all within a perceptual framework that aligns with human auditory processing.

Understanding these preprocessing steps is needed for debugging speech recognition systems (recognizing when normalization fails or when frame sizes are inappropriate), designing augmentation strategies (knowing which spectral regions to mask or shift), and adapting models to new acoustic environments (adjusting for different noise profiles or channel characteristics). The choices made in feature extraction, window sizes, and normalization strategies fundamentally constrain what the neural network can learn, making this pipeline a necessary foundation for all subsequent speech processing.