Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Negative Sampling › The Negative Sampling Objective

Michael BrenndoerferApril 4, 20259 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 21Negative Sampling › The Negative Sampling Objective

Reading section 4 of 15 · The Negative Sampling Objective · 9 minSection 4 of 15 · 9 min

Section 4 of 15 · 9 min read

Negative Sampling › The Negative Sampling ObjectiveOne section · Sections in this chapter4 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Having established that we can use sigmoid to score individual word pairs, we now face a key question: what exactly should the model optimize? The answer lies in constructing a clever objective function that captures our core intuition: observed context pairs should score high, while fake pairs should score low.

For each positive pair $(w_t, w_c)$ from the corpus, we sample $k$ negative words $\{w_{n_1}, w_{n_2}, \ldots, w_{n_k}\}$ from a noise distribution $P_n(w)$ . The objective for this training example combines both goals into a single expression:

\mathcal{L} = \log \sigma(\mathbf{w}'_c \cdot \mathbf{w}_t) + \sum_{i=1}^{k} \log \sigma(-\mathbf{w}'_{n_i} \cdot \mathbf{w}_t)

where:

$\mathcal{L}$ : objective function to maximize (higher is better)
$\mathbf{w}_t$ : center word embedding vector for word $w_t$
$\mathbf{w}'_c$ : context embedding vector for the positive context word $w_c$
$\mathbf{w}'_{n_i}$ : context embedding vector for the $i$ -th negative sample word
$k$ : number of negative samples per positive pair
$\sigma(\cdot)$ : sigmoid function
$\log \sigma(\mathbf{w}'_c \cdot \mathbf{w}_t)$ : positive term, maximized when the positive pair has high similarity
$\log \sigma(-\mathbf{w}'_{n_i} \cdot \mathbf{w}_t)$ : negative term, maximized when the negative pair has low similarity (negative dot product)

The positive term $\log \sigma(\mathbf{w}'_c \cdot \mathbf{w}_t)$ rewards the model for assigning high probability to observed pairs. When the dot product is large and positive, the sigmoid approaches 1, and $\log(1) = 0$ . When the dot product is small or negative, the sigmoid drops toward 0, and $\log(\text{small number})$ is a large negative penalty.

The negative terms $\sum_{i=1}^{k} \log \sigma(-\mathbf{w}'_{n_i} \cdot \mathbf{w}_t)$ reward the model for correctly identifying fake pairs. Notice the negative sign inside the sigmoid. For a negative pair to score well, we need $\sigma(-\text{dot product}) \approx 1$ , which happens when the dot product itself is negative (dissimilar embeddings). If the model mistakenly places a negative word close to the center word, the dot product becomes positive, the sigmoid drops, and the log becomes a large penalty.

Together, these terms create opposing forces: the first pulls observed context words toward the center word, while the second pushes random words away. Through millions of such updates, the embedding space self-organizes so that semantically related words cluster together.

In[9]:

Code

def negative_sampling_loss(center_vec, context_vec, negative_vecs):
    """
    Compute the negative sampling loss for one training example.

    Args:
        center_vec: Embedding of the center word
        context_vec: Embedding of the positive context word
        negative_vecs: List of embeddings for negative samples

    Returns:
        loss: The negative of the objective (to be minimized)
    """
    # Positive term: log sigma(context . center)
    pos_dot = np.dot(context_vec, center_vec)
    pos_loss = np.log(sigmoid(pos_dot) + 1e-10)

    # Negative terms: sum of log sigma(-negative . center)
    neg_loss = 0
    for neg_vec in negative_vecs:
        neg_dot = np.dot(neg_vec, center_vec)
        neg_loss += np.log(sigmoid(-neg_dot) + 1e-10)

    # We maximize the objective, so return negative for minimization
    return -(pos_loss + neg_loss)


# Example with random vectors
np.random.seed(42)
embedding_dim = 50
center = np.random.randn(embedding_dim) * 0.5
context = np.random.randn(embedding_dim) * 0.5
negatives = [np.random.randn(embedding_dim) * 0.5 for _ in range(5)]

loss = negative_sampling_loss(center, context, negatives)

Out[10]:

Console

Negative Sampling Loss Example:
--------------------------------------------------
Embedding dimension: 50
Number of negative samples: 5

Center . Context (positive): 1.0506
  sigma(dot) = 0.7409

Center . Negatives:
  Negative 1: dot = -1.3478, sigma(-dot) = 0.7938
  Negative 2: dot = 0.8562, sigma(-dot) = 0.2981
  Negative 3: dot = 3.7310, sigma(-dot) = 0.0234
  Negative 4: dot = 2.8612, sigma(-dot) = 0.0541
  Negative 5: dot = -1.9121, sigma(-dot) = 0.8713

Total loss (to minimize): 8.5505

With random initial embeddings, the dot products are near zero, producing sigmoid values around 0.5. The model is uncertain about all pairs. Training pushes these probabilities toward 1 for positive pairs and toward 0 for negatives.

Out[11]:

Visualization

Green line plot showing positive term loss decreasing from upper left to lower right. — Positive term loss as a function of dot product. The loss decreases as the dot product increases, rewarding the model for placing observed context words near the center word. The steep gradient near zero focuses learning on uncertain or incorrect predictions.

Red line plot showing negative term loss decreasing from upper right to lower left. — Positive term loss as a function of dot product. The loss decreases as the dot product increases, rewarding the model for placing observed context words near the center word. The steep gradient near zero focuses learning on uncertain or incorrect predictions.

The loss landscape reveals the asymmetric pressure applied by each term. For positive pairs (left), the loss drops sharply as the dot product increases, strongly encouraging the model to push observed context words closer. For negative pairs (right), the loss drops as the dot product becomes negative. The steepest gradients occur near zero, where the model is most uncertain.

Gradient ComputationLink Copied

The objective function creates a tug-of-war in embedding space. Each training step applies forces that reshape the geometry of word representations. Understanding these gradients reveals how semantic structure emerges from local updates.

During backpropagation, two types of forces act on each embedding. An attractive force from positive pairs pulls the center word embedding toward its true context word. A repulsive force from negative pairs pushes the center word away from each sampled negative. The magnitude of each force depends on the current prediction confidence. If the model already correctly predicts a positive pair, the gradient is small. If it is uncertain or wrong, the gradient is large, focusing learning where it matters most.

Starting from the objective function $\mathcal{L} = \log \sigma(\mathbf{w}'_c \cdot \mathbf{w}_t) + \sum_{i=1}^{k} \log \sigma(-\mathbf{w}'_{n_i} \cdot \mathbf{w}_t)$ , we compute gradients using the chain rule and the sigmoid derivative $\frac{d\sigma(x)}{dx} = \sigma(x)(1 - \sigma(x))$ .

Gradient for the center word embedding $\mathbf{w}_t$ :

\frac{\partial \mathcal{L}}{\partial \mathbf{w}_t} = (\sigma(\mathbf{w}'_c \cdot \mathbf{w}_t) - 1) \mathbf{w}'_c + \sum_{i=1}^{k} \sigma(\mathbf{w}'_{n_i} \cdot \mathbf{w}_t) \mathbf{w}'_{n_i}

where:

$(\sigma(\mathbf{w}'_c \cdot \mathbf{w}_t) - 1)$ : always negative since $\sigma(\cdot) \in (0, 1)$ , so this term pulls the center embedding toward the positive context
$\sigma(\mathbf{w}'_{n_i} \cdot \mathbf{w}_t)$ : the model's (mistaken) confidence that the $i$ -th negative pair comes from the corpus; during gradient ascent, this pushes the center embedding away from each negative

Gradient for the positive context word embedding $\mathbf{w}'_c$ :

\frac{\partial \mathcal{L}}{\partial \mathbf{w}'_c} = (\sigma(\mathbf{w}'_c \cdot \mathbf{w}_t) - 1) \mathbf{w}_t

Since the coefficient $(\sigma - 1)$ is negative, this gradient points toward the center word during gradient ascent, pulling the context embedding closer.

Gradient for each negative word embedding $\mathbf{w}'_{n_i}$ :

\frac{\partial \mathcal{L}}{\partial \mathbf{w}'_{n_i}} = \sigma(\mathbf{w}'_{n_i} \cdot \mathbf{w}_t) \mathbf{w}_t

The positive coefficient $\sigma(\cdot)$ means we subtract this during gradient ascent, pushing each negative embedding away from the center word. The key observation is that this gradient is proportional to how much the model currently confuses the negative for a real pair: if the model already assigns low probability to this negative pair, the gradient is small and the update is minor.

Out[12]:

Visualization

Line plot showing gradient magnitudes for positive and negative terms crossing at zero, with annotations marking high and low learning regions. — Gradient magnitude as a function of dot product for positive and negative terms. For positive pairs (green), the gradient |sigma - 1| is largest when the dot product is negative (wrong prediction) and smallest when positive (correct prediction). For negative pairs (red), the gradient |sigma| is largest when embeddings are incorrectly similar and smallest when already dissimilar. Both curves cross at zero where uncertainty is maximal, and the self-regulating behavior focuses learning effort on incorrect predictions.

The gradient curves cross at zero, where the model is maximally uncertain. For positive pairs, the model learns strongly when embeddings are dissimilar but relaxes when they are already similar. For negative pairs, the opposite holds. This elegant self-regulation emerges naturally from the sigmoid function.