Text-to-Speech › Non-Autoregressive Acoustic Modeling

While Tacotron 2 produces high-quality speech, its autoregressive nature prevents parallel generation. Each spectrogram frame must wait for the previous frame to be computed, resulting in synthesis speeds slower than real-time on standard hardware. This latency proves unacceptable for interactive applications like voice assistants, where users expect immediate responses, or for real-time captioning systems that must generate audio synchronized with live video.

Non-autoregressive TTS models, exemplified by FastSpeech and FastSpeech 2, eliminate sequential dependencies within the decoder, letting parallel generation of entire spectrograms. The key challenge is determining duration: how many acoustic frames correspond to each input phoneme? In autoregressive models, this duration emerges naturally from the attention alignment process. If the model attends to the phoneme "ae" (as in "cat") for 10 consecutive frames, that phoneme has a duration of 10 frames. Non-autoregressive models must explicitly model this duration mapping because they lack the sequential generation process that would otherwise reveal it.

In autoregressive models, the attention mechanism implicitly learns duration through the alignment process. If the model attends to the phoneme "ae" (as in "cat") for 10 consecutive frames, that phoneme has a duration of 10 frames. Non-autoregressive models must explicitly model this duration mapping because they cannot rely on the autoregressive generation process to reveal how long each phoneme should last.

FastSpeech 2 introduces a duration predictor, a small neural network trained to predict the length of each phoneme in frames. During inference, the model uses these predictions to expand the phoneme sequence through a length regulator: if phoneme $i$ has predicted duration $d_i$, it is repeated $d_i$ times along the time dimension before feeding into the spectrogram decoder. This expansion turns a sequence of linguistic features into a sequence of acoustic features with the appropriate temporal structure.

Mathematically, given encoder outputs $H = [h_1, h_2, ..., h_n]$ and durations $D = [d_1, d_2, ..., d_n]$, the length regulator constructs the expanded sequence:

$$H_{\text{expanded}} = [\underbrace{h_1, ..., h_1}_{d_1}, \underbrace{h_2, ..., h_2}_{d_2}, ..., \underbrace{h_n, ..., h_n}_{d_n}]$$

where:

• $H_{\text{expanded}}$: the expanded sequence of encoder outputs with each phoneme repeated according to its predicted duration
• $h_i$: the encoder output (hidden state) for the $i$-th phoneme
• $d_i$: the predicted duration (number of acoustic frames) for the $i$-th phoneme
• $n$: the total number of phonemes in the input sequence

The duration predictor is trained using Monotonic Alignment Search (MAS) or by extracting durations from a pretrained autoregressive teacher model. FastSpeech 2 also introduces variance predictors for pitch and energy (volume), letting fine-grained control over prosody without requiring explicit linguistic features. These variance predictors let the model to generate speech with varying intonation and stress, rather than creating flat, monotone output.

Just as the Transformer Architecture changed text processing, it has been adapted for TTS with architectures like Transformer TTS and its successor, the Neural Speech Synthesis models. These replace LSTM encoders and decoders with self-attention mechanisms. This makes possible better long-range dependencies for prosody modeling across phrases. Self-attention allows the model to directly relate distant phonemes, capturing coarticulation effects where the pronunciation of one sound is influenced by sounds several positions away. When generating the pitch contour for a multi-clause sentence, the model can relate the end of the first clause to the beginning of the second. This produces the gradual pitch declination characteristic of natural declarative speech.

However, pure self-attention lacks the inductive bias toward monotonic alignment inherent in recurrent models. Recurrent networks naturally process sequences in order, while self-attention treats all positions equally and symmetrically. This symmetry is desirable in text understanding but problematic in TTS, where strict left-to-right progress through the phoneme sequence is mandatory. Consequently, Transformer TTS systems often use positional encodings more aggressively or employ Relative Position Encodings to maintain sequential structure. Some variants augment the cross-attention with a diagonal attention prior, implemented as a Gaussian that penalizes large deviations from the expected monotonic diagonal, discouraging the attention mechanism from jumping erratically through the input.

The non-autoregressive variant FastSpeech uses a Feed-Forward Transformer (FFT) block consisting of self-attention and 1D convolutional layers, processing entire sequences in parallel once durations are determined. Each FFT block applies multi-head self-attention to capture global context, followed by two position-wise convolutional layers that model local spectral patterns. Stacking several FFT blocks in both the phoneme encoder and the mel-frame decoder creates a hierarchical representation: lower layers capture individual phoneme acoustics while higher layers integrate prosodic patterns over longer spans. This architecture generates spectrograms hundreds of times faster than autoregressive models while maintaining comparable quality, making real-time on-device synthesis practical.

FastSpeech 2 extends this framework by adding variance adaptors that condition generation on predicted pitch, energy (root-mean-square amplitude), and duration simultaneously. Rather than forcing the model to infer all prosodic dimensions implicitly from the text, these explicit predictors give fine-grained controllable handles. A user can increase the pitch predictor output to raise intonation, multiply the energy predictor to emphasize certain words, or scale the duration predictor to slow down delivery for clarity. This explicit decomposition also simplifies the training signal: each predictor has a dedicated supervised objective with ground-truth values extracted from reference audio, rather than relying solely on the spectrogram reconstruction loss to teach all prosodic properties at once.

The variance adaptor for pitch typically operates on the log-basic-frequency (log-F0) contour. Rather than predicting raw F0 in Hertz, the model predicts the logarithm of F0, which turns the typically right-skewed pitch distribution into a more Gaussian-shaped distribution easier for a neural network to model. During training, forced alignment tools extract ground-truth F0 values from reference recordings using pitch detection algorithms. At inference, the model predicts a normalized pitch shift per phoneme that is added to a speaker-specific mean pitch. This allows the system to transfer prosodic patterns from the training distribution while respecting the speaker's baseline range.