Speech Representations › From Waveforms to Spectra

Speech production generates pressure fluctuations that propagate through air as longitudinal waves. When these pressure variations reach a microphone, the transducer converts the mechanical energy of air pressure into electrical signals through electromagnetic or piezoelectric effects. To process these signals digitally, we must sample the continuous voltage waveform at regular intervals, converting instantaneous amplitude measurements into discrete numerical values that computers can manipulate.

The sampling theorem establishes the basic relationship between continuous and discrete representations of signals. To accurately capture a signal containing frequencies up to $f_{\max}$ without aliasing (distortion from undersampling), we must sample at a rate $f_s \geq 2f_{\max}$. This Nyquist rate ensures that the highest frequency component is represented by at least two samples per cycle, letting perfect reconstruction of the original continuous signal from its discrete samples.

If we sample too slowly, high-frequency components fold back into lower frequencies, creating phantom tones that corrupt the signal. Imagine trying to capture a spinning wheel with a strobe light: if the flashes come too slowly, the wheel appears to spin backward or stand still. Similarly, undersampled audio creates false frequencies that were never present in the original sound. This aliasing is not recoverable after the fact; once the signal has been sampled below the Nyquist rate, the original information is irretrievably lost.

The following visualization demonstrates this phenomenon with a 5 Hz sine wave. When sampled at 20 Hz (above the Nyquist rate of 10 Hz), the signal is captured accurately. When sampled at 4 Hz (below Nyquist), the signal appears as a 1 Hz alias, illustrating how undersampling folds high frequencies into lower ones.

For speech, most phonetic information lies below 8 kHz. The basic frequency of human voices typically ranges from 80 Hz (deep male voices) to 250 Hz (high female voices), with important formant structures extending up to around 7-8 kHz. This spectral content suggests a minimum sampling rate of 16 kHz for general-purpose speech recognition. Telephone systems historically used 8 kHz (capturing up to 4 kHz), which preserves intelligibility but loses the high-frequency consonant information that distinguishes sounds like "s" from "f." Modern ASR systems typically work with 16 kHz or 22.05 kHz. This provides enough bandwidth for clear phonetic discrimination while maintaining manageable data volumes. Music and high-fidelity audio extend to 44.1 kHz or 48 kHz to capture the full range of human hearing up to 20 kHz.

The raw waveform $x[n]$ represents amplitude at discrete time steps, but this representation is inefficient for phonetic analysis. The same phoneme spoken by different voices or in different contexts produces wildly different waveforms due to variations in pitch, speaking rate, and vocal tract anatomy, yet similar spectral patterns remain consistent. We need a representation that separates the slowly varying spectral envelope (carrying phonetic identity) from the rapid fine structure (carrying pitch and speaker characteristics). This separation allows recognition systems to identify phonemes regardless of whether they are spoken by a bass or soprano, while still preserving the acoustic cues that distinguish individual speakers.

Speech is inherently non-stationary, meaning its statistical properties change continuously over time as speakers move their articulators and transition between sounds. However, over short intervals of 20-30 milliseconds, the vocal tract configuration remains approximately constant, and the signal behaves locally like a stationary process. This quasi-stationarity arises because human articulators, such as the tongue and lips, have mass and inertia that prevent instantaneous changes. A speaker transitioning from "ee" to "ah" does not flip a switch; instead, the tongue body moves over tens to hundreds of milliseconds, creating a gradual spectral trajectory that frames can track.

The Short-Time Fourier Transform (STFT) exploits this property by applying the Fourier transform to overlapping windows of the signal, analyzing each frame as if it were a stationary tone complex.

The STFT computes:

$$X[m, \omega] = \sum_{n=-\infty}^{\infty} x[n] \cdot w[n-m] \cdot e^{-j\omega n}$$

where:

• $X[m, \omega]$: the complex-valued STFT coefficient at time frame $m$ and angular frequency $\omega$, representing both magnitude and phase
• $x[n]$: the discrete-time input signal at sample index $n$
• $w[n-m]$: the window function centered at time $m$, which tapers the signal to zero at frame boundaries to prevent spectral leakage
• $j$: the imaginary unit ($j = \sqrt{-1}$), representing the 90-degree phase shift in complex exponentials
• $e^{-j\omega n}$: the complex exponential basis function (cosine and sine components) for frequency analysis
• $m$: the time frame index, showing the position of the analysis window
• $\omega$: the angular frequency in radians per sample
• $n$: the sample index within the summation window, ranging over the effective window length

The window function $w[n-m]$ is centered at time $m$ and tapers the signal to zero at the edges, preventing spectral leakage. Without this tapering, the abrupt truncation of the signal at frame boundaries would create discontinuities, introducing high-frequency artifacts that spread across the spectrum and obscure the true frequency content. The window slides across the signal with a hop length typically set to 25% or 50% of the window size to ensure continuity between frames and prevent temporal aliasing.

The choice of window length creates an unavoidable tradeoff between time and frequency resolution, sometimes called the Heisenberg-Gabor limit in signal processing. A longer window improves frequency resolution (finer frequency bins) but reduces temporal resolution (each frame covers a longer time span, blurring rapid changes). A shorter window captures transient events precisely in time but cannot resolve closely spaced frequency components. For speech recognition, the standard 25ms window strikes a balance: long enough to resolve formant frequencies that are typically spaced 200-400 Hz apart, yet short enough to capture the rapid spectral changes of stop consonants and fricatives that evolve over 20-50ms.

Common window functions include:

• Hamming window: $w[n] = 0.54 - 0.46 \cos\!\left(\frac{2\pi n}{N-1}\right)$
• Hann window: $w[n] = 0.5 - 0.5 \cos\!\left(\frac{2\pi n}{N-1}\right)$

The Hamming window minimizes the maximum side lobe, reducing spectral leakage at the cost of slightly wider main lobes. The Hann window offers smoother spectral decay with lower side lobes but slightly worse frequency resolution. For speech, the Hamming window has been traditional due to its optimal side-lobe properties, though Hann and Blackman windows also see use in modern systems where reduced spectral leakage is prioritized.

@fig-window-functions compares these window functions in both time and frequency domains. The time domain shows the tapering applied to frame edges, while the frequency domain reveals the tradeoff between main lobe width (frequency resolution) and side lobe levels (spectral leakage).

The output of the STFT is a complex matrix representing magnitude and phase for each time-frequency bin. For speech recognition, we typically discard the phase and work with the power spectrum or magnitude spectrum, as human perception is relatively insensitive to phase relationships between frequencies. This phase insensitivity, sometimes called "phase deafness," arises from the way the cochlea processes sound: hair cells respond to the amplitude envelope of frequency bands rather than instantaneous phase. The squared magnitude of each complex coefficient gives us the power spectrum:

$$|X[m, k]|^2 = \text{Re}(X[m, k])^2 + \text{Im}(X[m, k])^2$$

where:

• $|X[m, k]|^2$: the power spectral density (energy) at time frame $m$ and frequency bin $k$
• $\text{Re}(X[m, k])$: the real component of the complex STFT coefficient
• $\text{Im}(X[m, k])$: the imaginary component of the complex STFT coefficient
• $m$: the time frame index
• $k$: the frequency bin index, corresponding to discrete frequencies from 0 to the Nyquist rate

Discarding phase simplifies the representation while retaining the phonetically relevant information for recognition. The tradeoff matters primarily when reconstructing audio from features (as in speech synthesis or enhancement), where phase recovery algorithms like Griffin-Lim must be applied.

Visualizing the power spectrum across time produces the spectrogram, a two-dimensional heatmap showing how frequency content evolves throughout an utterance. The horizontal axis represents time, the vertical axis represents frequency, and color intensity represents energy (typically in decibels). This representation provides an intuitive view of speech structure, revealing horizontal bands of energy corresponding to formants, vertical striations showing periodic glottal pulses, and transient bursts marking consonant releases.

Standard spectrograms use a linear frequency scale, equally spacing Hz across the vertical axis. This allocation misaligns with human perception. We hear pitch logarithmically: the perceptual distance between 100 Hz and 200 Hz is much larger than between 1000 Hz and 1100 Hz, even though both differ by 100 Hz. Our frequency resolution decreases as frequency increases, following the organization of the basilar membrane in the cochlea. A linear spectrogram wastes resolution on high frequencies where we cannot discriminate fine differences, while undersampling the low frequencies where small changes matter perceptually. For instance, the distinction between the first formants of "bat" and "bet" occurs in the 500-700 Hz range, requiring fine resolution that linear spectrograms spread thin across the entire frequency axis.

A related concept is the distinction between "narrowband" and "broadband" spectrograms, which reflect different window length choices rather than frequency axis scaling. A narrowband spectrogram uses a long window (e.g., 100ms), creating fine frequency resolution that reveals individual harmonics as horizontal striations. A broadband spectrogram uses a short window (e.g., 5ms), sacrificing frequency resolution to reveal rapid temporal events like glottal pulses as vertical striations. Modern ASR systems typically use configurations closer to broadband, focusing on the spectral envelope rather than individual harmonics, since the harmonic spacing carries pitch information that is not reliably correlated with phonetic identity across speakers.