LSTM Gate Equations › The Input Gate

While the forget gate decides what to discard, the input gate addresses the complementary question: what new information deserves to be written into memory? Not every input is equally important. When reading text, some words determine the meaning ("The murderer was...") while others are grammatical scaffolding ("the", "was"). The input gate learns to recognize which incoming information warrants permanent storage.

The input gate works in tandem with a candidate cell state (which we'll examine next). Think of it as a two-stage process: the candidate proposes "here's what we could store," and the input gate decides "here's how much of that proposal to actually write." This separation of concerns allows the network to generate rich candidate updates while maintaining fine-grained control over what enters long-term memory.

Like the forget gate, the input gate examines the current input and previous hidden state to make context-dependent decisions:

where:

• $\mathbf{i}_t \in \mathbb{R}^h$: the input gate activation vector, controlling write intensity per dimension
• $\mathbf{W}_i \in \mathbb{R}^{h \times d}$: weight matrix that learns which input features signal "important, store this"
• $\mathbf{U}_i \in \mathbb{R}^{h \times h}$: weight matrix that learns which processing contexts warrant new storage
• $\mathbf{b}_i \in \mathbb{R}^h$: bias vector setting the default write behavior

The structure is identical to the forget gate: a linear combination of the input and hidden state, followed by sigmoid. The difference lies entirely in the learned parameters. During training, the network discovers that certain input patterns should trigger forgetting while different patterns should trigger storage. The same architectural template serves opposite purposes through different learned weights.