LSTM Gate Equations › Complete LSTM Equations

Michael BrenndoerferUpdated May 11, 20251 min read
Language AI HandbookAll chapters
Part 11 of 67Recurrent Neural Networks
Chapter 78LSTM Gate Equations › Complete LSTM Equations
Reading section 9 of 17 · Complete LSTM Equations · 1 minSection 9 of 17 · 1 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 78 of 513

Section 9 of 17 · 1 min read

LSTM Gate Equations › Complete LSTM EquationsOne section · Sections in this chapter9 / 17 ↓
Sections in this chapter17
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's collect all the equations in one place for reference. Given inputs xt\mathbf{x}_t, ht−1\mathbf{h}_{t-1}, and ct−1\mathbf{c}_{t-1}, the LSTM computes:

ft=σ(Wfxt+Ufht−1+bf)(forget gate)it=σ(Wixt+Uiht−1+bi)(input gate)c~t=tanh⁡(Wcxt+Ucht−1+bc)(candidate)ct=ft⊙ct−1+it⊙c~t(cell state update)ot=σ(Woxt+Uoht−1+bo)(output gate)ht=ot⊙tanh⁡(ct)(hidden state)\begin{aligned} \mathbf{f}_t &= \sigma(\mathbf{W}_f \mathbf{x}_t + \mathbf{U}_f \mathbf{h}_{t-1} + \mathbf{b}_f) & \text{(forget gate)} \\ \mathbf{i}_t &= \sigma(\mathbf{W}_i \mathbf{x}_t + \mathbf{U}_i \mathbf{h}_{t-1} + \mathbf{b}_i) & \text{(input gate)} \\ \tilde{\mathbf{c}}_t &= \tanh(\mathbf{W}_c \mathbf{x}_t + \mathbf{U}_c \mathbf{h}_{t-1} + \mathbf{b}_c) & \text{(candidate)} \\ \mathbf{c}_t &= \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t & \text{(cell state update)} \\ \mathbf{o}_t &= \sigma(\mathbf{W}_o \mathbf{x}_t + \mathbf{U}_o \mathbf{h}_{t-1} + \mathbf{b}_o) & \text{(output gate)} \\ \mathbf{h}_t &= \mathbf{o}_t \odot \tanh(\mathbf{c}_t) & \text{(hidden state)} \end{aligned}

These six equations fully specify the LSTM cell. Notice the symmetry: three gates (forget, input, output) all have the same structure, differing only in their learned parameters. The candidate has the same structure but uses tanh⁡\tanh instead of sigmoid. The cell state and hidden state updates are simple element-wise operations.