RNN Architecture › The Problem with Feedforward Networks on Sequences

Part X covered feedforward networks and the building blocks of deep learning. Those architectures have a fixed input size. You feed in a vector, and the network produces an output. To handle multiple inputs, you stack layers, but each forward pass is stateless: no information persists between calls. The network does not remember what it processed one second ago.

Consider trying to predict the next word in a sentence. The word "bank" in "She walked to the river bank" means something completely different from "bank" in "She deposited money at the bank." Context resolves ambiguity. A feedforward network processing only a single-word window would see "bank" in isolation and have no mechanism to access the surrounding context. You could widen the window, feeding in five words at once, but this forces a fixed context length and creates a different input structure for every window size. Worse, it requires a separate set of parameters for every position in the window: the weights for the first word slot are entirely independent of the weights for the fifth word slot. The network cannot share what it learns about word patterns across positions.

This position-specific parameterization means the model must learn the same linguistic patterns independently for every context slot. If it learns that a certain word appearing near the start of a sentence suggests a certain topic, that knowledge does not automatically transfer to predicting topic when that word appears in the middle of the sentence. This is a fundamental mismatch between how language works and how fixed-window feedforward networks represent it.

RNNs avoid this limitation by processing sequences step by step and maintaining a running summary. Rather than requiring a fixed-size window, they thread context through a hidden state that grows richer as more of the sequence is consumed. The network architecture stays the same regardless of sequence length; only the number of recurrent steps changes. A sentence of 5 words and a paragraph of 500 words go through the same model, just for different numbers of steps.

The idea of maintaining state across computations has roots in control theory and classical automata theory. A finite state machine reads a sequence of symbols and transitions between states at each step, with the current state encoding everything relevant from the past. An RNN is a continuous, parameterized, differentiable version of this idea: instead of a discrete state from a finite set, it maintains a continuous vector state learned through gradient descent.