RNN Architecture › Parameter Sharing Across Time Steps

One of the most powerful aspects of the RNN is that the same parameters are reused at every time step. Unlike a feedforward network with separate weights for each position, the RNN applies the same transformations $\mathbf{W}_{hh}$, $\mathbf{W}_{xh}$, and $\mathbf{W}_{hy}$ regardless of where in the sequence we are.

This parameter sharing has several important consequences:

• Efficiency: The model stays compact regardless of sequence length. A sequence of 100 tokens and a sequence of 10,000 tokens use the same parameters.
• Generalization across positions: A pattern learned at position 5 generalizes to position 500 automatically. The network does not need to re-learn the same patterns for every possible position.
• Length generalization: At test time, the RNN can process sequences longer than any it saw during training, because the weights apply the same computation at every step.
• Translation invariance in time: If you shift the entire sequence by one time step, the model's behavior for each element is unchanged. This mirrors the translation invariance that convolutional networks achieve in the spatial domain, except it applies to the temporal domain.

This is in contrast to a "wide" feedforward approach where you concatenate all token embeddings into a single input vector. That approach requires a separate set of weights for every position, does not generalize across positions, and cannot handle variable-length inputs without padding to the maximum possible length.

The parameter sharing is also what makes RNNs well-suited to streaming applications. Because the model applies the same computation at every step, it can process an online stream of tokens one by one without needing to buffer the entire sequence first. Each new token updates the hidden state, and a prediction can be made immediately after the update.

The parameter sharing in RNNs has a deep theoretical connection to computation. A Turing machine executes the same finite set of rules (a program) at every step, acting on the current tape symbol and machine state to produce a new state and move the tape head. An RNN is structurally similar: it applies the same transformation at every step, taking the current input and hidden state to produce a new hidden state. This analogy motivates the theoretical result (proved by Siegelmann and Sontag in 1992) that RNNs with enough hidden units and rational weights are Turing-complete: they can in principle simulate any computation. Whether practical finite-precision RNNs trained by gradient descent can learn complex programs is a separate empirical question, but the theoretical equivalence is conceptually illuminating.