RNN Architecture › The Recurrent Connection Intuition

The key innovation in an RNN is the recurrent connection: the hidden state at time step $t$ depends on the current input $\mathbf{x}_t$ and on the hidden state from the previous time step $\mathbf{h}_{t-1}$.

Think of it as reading a book. Each time you finish a sentence, you carry a mental summary forward. That summary informs how you interpret the next sentence. You do not re-read the entire book to understand each new sentence; instead, you rely on the compressed representation in your memory. The hidden state plays exactly this role.

The hidden state is not directly observed or directly supervised. It is an internal representation that the training process shapes to be useful for the downstream prediction task. If the task is language modeling, the hidden state will learn to encode information about grammatical structure, topic, and recent word history. If the task is sentiment classification, the hidden state will accumulate evidence about the emotional tone of the text.

Concretely, the recurrent connection looks like a cycle in the network graph. The output of the hidden layer feeds back into itself at the next time step. This single cycle is what separates an RNN from a feedforward network and what allows it to process sequences of arbitrary length. The cycle is mathematically well-defined because the flow is across time: at step $t$ the network reads $\mathbf{h}_{t-1}$ (already computed) and produces $\mathbf{h}_t$ (the new output). There is no ambiguous circular dependency within a single time step.

The recurrent connection also means the network's behavior is dynamic. For the same current input $\mathbf{x}_t$, the network will produce a different hidden state depending on what it has seen before. The word "not" followed by "happy" should drive a different hidden state than "not" followed by "guilty", and the recurrent weights learn to encode exactly this kind of history-sensitive behavior.