GloVe › Bias Terms in GloVe

The bias terms $b_i$ and $\tilde{b}_j$ play an important and often underappreciated role in making the objective symmetric and absorbing word-frequency effects.

Recall that the derivation gave us $\vec{w}_i^T \tilde{\vec{w}}_j = \log X_{ij} - \log X_i$. The term $\log X_i$ (the log total context count for word $i$) depends only on how frequent word $i$ is overall. Very frequent words like "the" have large $X_i$ regardless of the specific context word. If we did not include a bias term, this frequency effect would be absorbed into the dot product, corrupting the relational structure the vectors are supposed to capture. The bias $b_i$ absorbs this word-frequency effect, freeing the dot product to represent purely relational information.

Similarly, $\tilde{b}_j$ absorbs the frequency effect for word $j$ in its role as a context word. Together, $b_i + \tilde{b}_j$ accounts for both the target and context word's marginal frequencies, letting the dot product $\vec{w}_i^T \tilde{\vec{w}}_j$ focus on the residual: how much more or less often $i$ and $j$ co-occur than we would predict from their individual frequencies alone. This residual is precisely what PMI measures, as we will see in the next section.

The bias terms are simple scalars, so they add very few parameters (only $2V$ additional values) while providing a significant modeling benefit. In practice, the biases converge quickly during training because they are solving a simple regression problem: fit the word's average log-count across all its context partners.

An elegant consequence of the bias terms is that the objective behaves consistently when we swap the roles of target and context words. In the co-occurrence matrix $\mathbf{X}$, entry $X_{ij}$ counts word $j$ in word $i$'s context, and $X_{ji}$ counts word $i$ in word $j$'s context. For symmetric windows, $X_{ij} = X_{ji}$. The GloVe objective with bias terms naturally accommodates this symmetry: both $(\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$ and $(\vec{w}_j^T \tilde{\vec{w}}_i + b_j + \tilde{b}_i - \log X_{ji})^2$ are minimized simultaneously.

GloVe's symmetric design means that after training, the word vector $\vec{w}_i$ and its context vector $\tilde{\vec{w}}_i$ capture similar but not identical information. They have learned from the same co-occurrence data, just from different roles in the objective. A common and empirically validated practice is to use $\vec{w}_i + \tilde{\vec{w}}_i$ as the final word representation, combining the two complementary views for slightly improved performance. This is analogous to ensemble averaging: the two vectors have seen the same data from different angles, and combining them reduces variance in the final representation.

Why do the two vectors differ at all if they see the same data? Because the objective is not symmetric in the parameters, even though it is symmetric in the data. The gradients with respect to $\vec{w}_i$ depend on $\tilde{\vec{w}}_j$, and vice versa. So although both vectors converge toward representations that encode the same underlying statistics, the optimization path is different, and the resulting vectors are slightly different directions in the embedding space. Summing them gives a representation that combines both views.