Text-to-Speech › Sequence-to-Sequence Acoustic Modeling

The dominant paradigm for acoustic modeling treats TTS as a sequence-to-sequence translation problem, mapping from a sequence of characters or phonemes to a sequence of mel-spectrogram frames. This approach, pioneered by Tacotron, uses the attention mechanisms we studied in Part XII to handle the length discrepancy between text and audio. Unlike machine translation where source and target sequences might be similar in length, TTS involves significant expansion: a single phoneme might generate dozens of acoustic frames, requiring the model to learn temporal allocation.

Tacotron 2 exemplifies the encoder-decoder approach to TTS. The architecture consists of five components working in concert:

1. Encoder: Converts input character or phoneme sequences into hidden representations that capture linguistic context
2. Attention mechanism: Aligns encoder states with decoder time steps, determining which input characters to focus on when generating each acoustic frame
3. Decoder: Autoregressively predicts mel-spectrogram frames one at a time, conditioning each prediction on previously generated frames
4. Post-net: Refines decoder outputs with residual convolutions to capture fine spectral details and reduce artifacts
5. Stop token prediction: Determines when to halt generation dynamically based on the content

The encoder processes input characters through a convolutional pre-net followed by a bidirectional LSTM. As we discussed in Part XI: Bidirectional RNNs, bidirectional processing ensures that each character's representation incorporates both left and right context, which is needed for determining pronunciation from surrounding letters. For example, the "g" in "great" requires looking ahead to the "r" to know it should be pronounced as a voiced velar stop rather than the voiced palato-alveolar affricate found in "genre."

The decoder operates autoregressively, similar to the language models in Part XXVIII. At each time step $t$, the decoder receives the mel-spectrogram frame predicted at step $t-1$ (or the ground truth during training via Teacher Forcing), processes it through a pre-net and feeds it into an autoregressive LSTM. The LSTM hidden state queries the encoder outputs through an attention mechanism to determine which input characters to focus on when generating the current acoustic frame. This autoregressive structure ensures temporal coherence in the generated spectrogram, as each frame builds upon the acoustic context established by previous frames.

Standard Bahdanau Attention computes alignment scores based solely on content matching between query and key vectors. However, TTS requires monotonic alignment: the model should attend to input characters in left-to-right order without skipping or repeating. Speech is strictly sequential; unlike machine translation where distant word pairs might align due to grammatical reordering, acoustic features must follow the phonetic sequence exactly. If the model attends to the final phoneme while still generating audio for the first phoneme, the result is unintelligible babbling.

Tacotron 2 employs location-sensitive attention, which augments content-based scoring with location information. The alignment score for decoder step $i$ attending to encoder step $j$ incorporates the previous alignment weights:

$$e_{i,j} = v^T \tanh(W_q h_i + W_k h_j + W_f f_{i,j} + b)$$

where:

• $e_{i,j}$: the alignment score (energy) for decoder step $i$ attending to encoder step $j$
• $v$: a learned weight vector for computing the attention score
• $W_q$: weight matrix for the decoder query (hidden state $h_i$)
• $h_i$: the hidden state of the decoder at step $i$
• $W_k$: weight matrix for the encoder key (hidden state $h_j$)
• $h_j$: the hidden state of the encoder at position $j$ (the value being attended to)
• $W_f$: weight matrix for the location features
• $f_{i,j}$: the convolutional features of the previous alignment vector at position $j$
• $b$: bias term

By convolving the previous attention weights with learned filters, the model can sense whether it has been attending to the left or right of the current position, encouraging forward movement through the text. This prevents the "babbling" artifacts that occur when attention gets stuck oscillating between two input positions or jumps erratically through the sequence. The convolutional features give a sense of momentum. This keeps once the model starts processing a particular phoneme, it continues to generate the appropriate number of acoustic frames before moving to the next.

The attention weights are normalized across the input sequence using a softmax, creating a context vector:

$$c_i = \sum_j \alpha_{i,j} h_j$$

where:

• $c_i$: the context vector at decoder step $i$, representing a weighted sum of relevant encoder states
• $\alpha_{i,j}$: the attention weight showing the probability (between 0 and 1) of attending to encoder position $j$ when generating decoder output $i$
• $h_j$: the hidden state of the encoder at position $j$ (the value being attended to)
• $j$: the index iterating over all encoder positions

The attention weights are computed via softmax normalization:

$$\alpha_{i,j} = \frac{\exp(e_{i,j})}{\sum_k \exp(e_{i,k})}$$

where:

• $\alpha_{i,j}$: the normalized attention weight for decoder step $i$ and encoder step $j$, computed as the softmax of alignment scores
• $e_{i,j}$: the alignment score (energy) for encoder position $j$ used in the numerator
• $e_{i,k}$: the alignment score for encoder position $k$ used in the denominator summation
• $k$: the index iterating over all encoder positions for normalization (so $\sum_j \alpha_{i,j} = 1$)

The softmax ensures that the attention weights form a valid probability distribution across the input sequence, letting the model to focus on specific phonemes while generating each acoustic frame. This context vector, concatenated with the decoder LSTM output, predicts the mel-spectrogram frame through a linear projection, effectively conditioning the acoustic output on the relevant linguistic content.

Unlike text generation where we typically generate fixed-length sequences or use special end-of-sequence tokens, TTS must dynamically determine when speech concludes. Different utterances have different lengths, and the model must learn to recognize when it has generated sufficient acoustic frames to stand for the complete input text. Tacotron 2 adds a binary classification head predicting a "stop token" probability. When this probability exceeds a threshold (typically 0.5), generation terminates.

During training, the model minimizes two losses simultaneously:

$$\mathcal{L}_{\text{TTS}} = \mathcal{L}_{\text{mel}} + \mathcal{L}_{\text{stop}}$$

where:

• $\mathcal{L}_{\text{TTS}}$: the total loss function for the TTS model, combining reconstruction and termination objectives
• $\mathcal{L}_{\text{mel}}$: the mel-spectrogram reconstruction loss (typically L1 or L2 distance), measuring how well the predicted spectrogram matches the ground truth
• $\mathcal{L}_{\text{stop}}$: the stop token prediction loss (binary cross-entropy), determining when the model should halt generation

The model optimizes both objectives simultaneously. This keeps accurate spectral prediction while learning appropriate stopping criteria for variable-length utterances. The mel loss is typically the L1 or L2 distance between predicted and ground-truth spectrograms, capturing the fine details of the spectral envelope, while the stop token loss uses binary cross-entropy. Because the large majority of frames are non-terminal (only the final frame should predict stop), the stop token loss is usually weighted to account for class imbalance. This keeps the model does not just learn to never predict termination.