Time Delay Neural Networks (1987) › How TDNNs Work

To make these concepts concrete, let's trace through exactly how a TDNN processes a speech signal. Imagine we're trying to recognize the word "cat." After preprocessing the audio, we have a sequence of feature vectors, each representing the acoustic properties at a particular moment. Perhaps we have five time steps worth of features: $f_0, f_1, f_2, f_3, f_4$. The first few feature vectors capture the initial "k" sound with its burst of energy across specific frequency bands. The middle vectors represent the vowel "a" with its characteristic formant structure. The final vectors encode the "t" sound with its brief stop and release.

The TDNN processes this sequence by sliding a window across time. Suppose we use a window size of three, meaning the network examines three consecutive time steps together. At the first position, the window sees $[f_0, f_1, f_2]$, capturing the transition from the "k" onset through the beginning of the vowel. The network applies its learned weights to these three feature vectors, computing a weighted combination and passing it through an activation function to produce a hidden state $h_1$.

The window then slides forward one step. Now it examines $[f_1, f_2, f_3]$, spanning from the middle of the "k" through the heart of the "a" vowel. The network applies exactly the same weights $W$ to this window that it applied to the first position. This weight sharing means the network's learned feature detectors operate identically at every temporal position. The computation produces hidden state $h_2$: applying weights to the windowed input, adding a bias term, and passing through the activation function.

The process continues as the window slides across the entire sequence. At the third position, examining $[f_2, f_3, f_4]$, the same weights produce hidden state $h_3$. Each of these hidden states represents what the network has detected in its local temporal window. Because the weights are shared, a detector that learns to recognize formant transitions will fire whenever it encounters that pattern, regardless of whether it appears early or late in the sequence. This is precisely what we want: the acoustic signature of a phoneme shouldn't depend on when it occurs in an utterance.

We can express the temporal convolution operation more formally. At each time position $t$, the network computes a hidden state $h_t$ by applying its weights to the windowed input:

$h_t = \sigma\left(\sum_{i=0}^{k-1} W_i \cdot x_{t+i} + b\right)$

This equation captures the entire sliding window process. The sum runs from $i = 0$ to $k-1$, where $k$ is the window size, examining $k$ consecutive time steps. For each position $i$ within the window, we have a weight matrix $W_i$ that multiplies the input features $x_{t+i}$ from that time step. The subscript $t+i$ reflects how we're looking at a sequence of inputs starting at time $t$. After computing the weighted sum across the window and adding a bias term $b$, we apply an activation function $\sigma$ to introduce nonlinearity.

The beauty of this formulation is that the same weights $W_i$ are used at every temporal position $t$. Whether we're computing $h_1, h_2,$ or $h_{100}$, we use the same learned parameters. This weight sharing across time is what makes the network efficient and generalizable.

In a multi-layer TDNN, we stack these temporal convolutions to build hierarchical representations. The output of one layer becomes the input to the next, allowing the network to learn increasingly abstract temporal features:

$h_t^{(l+1)} = \sigma\left(\sum_{i=0}^{k-1} W_i^{(l)} \cdot h_{t+i}^{(l)} + b^{(l)}\right)$

Here the superscript $(l)$ denotes the layer. The hidden states from layer $l$ serve as inputs to layer $l+1$, which applies its own temporal convolution with its own learned weights $W_i^{(l)}$. This stacking enables the network to capture temporal patterns at multiple timescales. Early layers might detect brief acoustic events lasting a few milliseconds, while deeper layers combine these into phoneme representations spanning tens of milliseconds.

Consider the practical task of recognizing the phoneme "k" at the beginning of "cat." The acoustic signature of this plosive consonant involves a brief silence followed by a burst of energy concentrated in specific frequency bands. In the input features $[f_0, f_1, f_2, f_3, f_4]$ representing the word, this pattern likely appears in the first two or three time steps.

As the TDNN's sliding window moves across the sequence, different hidden units in the network activate in response to different acoustic events. When the window covers the "k" burst at the beginning, certain hidden units trained to detect plosive onsets will activate strongly. As the window slides to encompass the vowel "a," different units tuned to formant patterns characteristic of vowels will fire. The network builds up a representation of the entire acoustic sequence through these overlapping temporal analyses.

During training, the network learns which combinations of acoustic patterns across its temporal window correspond to which phonemes. Because the weights are shared across time, the network doesn't learn that "k" sounds specifically occur at the beginning of sequences. Instead, it learns the general acoustic pattern of "k" sounds, which it can then recognize whether they appear in "cat," "kept," "take," or any other context. This generalization across temporal position is what makes TDNNs effective for speech recognition with realistic variability in timing and speaking rate.