Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GloVe › The Weighting Function

Michael BrenndoerferApril 10, 20254 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 25GloVe › The Weighting Function

Reading section 4 of 13 · The Weighting Function · 4 minSection 4 of 13 · 4 min

Section 4 of 13 · 4 min read

GloVe › The Weighting FunctionOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The weighting function $f$ must satisfy the properties listed above. The original GloVe paper proposes a specific polynomial form with a plateau:

f(x) = \begin{cases} (x/x_{\max})^\alpha & \text{if } x < x_{\max} \\ 1 & \text{otherwise} \end{cases}

where:

$x$ : the co-occurrence count $X_{ij}$ for the pair being weighted
$x_{\max}$ : the saturation threshold at which weight reaches its maximum of 1 (default: 100)
$\alpha$ : the exponent controlling how quickly weight grows with count (default: $3/4$ )

Let us unpack each design choice carefully, because together they handle several failure modes at once.

The saturation threshold $x_{\max}$ : Co-occurrences above $x_{\max}$ receive full weight 1. Below $x_{\max}$ , weight scales polynomially from 0 to 1. This prevents very common co-occurrences, like "the" appearing with essentially every content word, from dominating the objective simply by virtue of their frequency. Without this cap, function words would drown out the signal from semantically rich but less frequent pairs. With the cap, every pair above $x_{\max}$ contributes equally to the objective, and only pairs below the threshold get downweighted.

The exponent $\alpha = 3/4$ : This value is not arbitrary. It is the same exponent used in Word2Vec's negative sampling unigram distribution. Values close to 1 give nearly linear weighting (doubling the count roughly doubles the weight), while values close to 0 give more uniform weighting (the weight barely changes with count). The $3/4$ exponent is a middle ground: it penalizes very rare co-occurrences without being too aggressive. Empirically, it produces better embeddings than either $\alpha = 1$ (too linear) or $\alpha = 0.5$ (too flat). The $3/4$ value was tuned in the original paper across multiple downstream tasks.

Zero weight for zero co-occurrences: Because $f(0) = 0$ , pairs where $X_{ij} = 0$ never appear in the sum. This elegantly sidesteps the $\log 0$ problem. You do not need to add a smoothing constant or handle zeros as a special case; the weighting function simply ignores them. In practice, the sum runs over the nonzero entries of $\mathbf{X}$ , which is exactly the set of pairs that contain real information.

One subtlety: the choice of $x_{\max} = 100$ means that only pairs co-occurring at least 100 times receive full weight. For smaller corpora, you may need to lower this threshold. If your corpus has fewer than a million tokens, a threshold of 10 or 20 may be more appropriate.

In[6]:

Code

import numpy as np


def glove_weight(x, x_max=100, alpha=0.75):
    """GloVe weighting function f(x)."""
    return np.where(x < x_max, (x / x_max) ** alpha, 1.0)


x = np.linspace(0, 200, 500)
w = glove_weight(x)

Out[7]:

Visualization

Line plot of GloVe weighting function showing polynomial growth below 100 and a flat saturation line at weight 1 above. — GloVe weighting function f(x) with x_max=100 and alpha=0.75. Co-occurrences above 100 receive full weight 1, while rarer pairs receive fractional weight proportional to (x/100)^0.75. The polynomial growth prevents very rare co-occurrences from contributing too much noise, and zero co-occurrences receive zero weight so log(0) never appears in the objective. The shaded area highlights the transition region from zero to full weight.

The weighting function makes GloVe robust to two failure modes: rare co-occurrences that are dominated by noise, and very common co-occurrences that tend to reflect syntactic accidents rather than semantic relationships. The polynomial shape below $x_{\max}$ means the transition from zero weight to full weight is smooth, which helps with gradient-based optimization.