Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Negative Sampling › From Softmax to Sigmoid

Michael BrenndoerferApril 4, 20254 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 21Negative Sampling › From Softmax to Sigmoid

Reading section 3 of 15 · From Softmax to Sigmoid · 4 minSection 3 of 15 · 4 min

Section 3 of 15 · 4 min read

Negative Sampling › From Softmax to SigmoidOne section · Sections in this chapter3 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The mathematical reformulation is straightforward. Instead of softmax, we use the sigmoid function $\sigma(x) = 1 / (1 + e^{-x})$ to output a probability for each pair independently.

For a positive (center, context) pair, we want the model to output a high probability that this pair comes from the corpus. The sigmoid function maps the dot product of the embeddings to a probability:

P(D = 1 | w_c, w_t) = \sigma(\mathbf{w}'_c \cdot \mathbf{w}_t) = \frac{1}{1 + \exp(-\mathbf{w}'_c \cdot \mathbf{w}_t)}

where:

$P(D = 1 | w_c, w_t)$ : probability that the word pair $(w_c, w_t)$ is a observed context pair from the data
$D$ : binary indicator variable (1 = observed pair from data, 0 = fake pair)
$\sigma(x)$ : sigmoid function, mapping any real number to the range $(0, 1)$
$\mathbf{w}'_c$ : context embedding vector for word $w_c$
$\mathbf{w}_t$ : center embedding vector for word $w_t$
$\mathbf{w}'_c \cdot \mathbf{w}_t$ : dot product between the context and center embeddings, measuring their similarity

When the embeddings are similar (large positive dot product), the sigmoid output approaches 1, indicating high confidence that the pair comes from the corpus.

For negative pairs, we want $P(D = 1) \approx 0$ , or equivalently $P(D = 0) \approx 1$ . Using the property that $1 - \sigma(x) = \sigma(-x)$ , we write:

P(D = 0 | w_n, w_t) = 1 - \sigma(\mathbf{w}'_n \cdot \mathbf{w}_t) = \sigma(-\mathbf{w}'_n \cdot \mathbf{w}_t)

where:

$w_n$ : a negative (randomly sampled) word that did not appear in the context
$\mathbf{w}'_n$ : context embedding vector for the negative word $w_n$

When the embeddings are dissimilar (negative dot product), $\sigma(-\mathbf{w}'_n \cdot \mathbf{w}_t)$ approaches 1, correctly indicating that the pair is likely fake.

In[6]:

Code

def sigmoid(x):
    """Compute sigmoid function with numerical stability."""
    x = np.clip(x, -500, 500)
    return 1 / (1 + np.exp(-x))


# Example: probability computation for word pairs
dot_product_positive = 2.5  # High similarity (learned)
dot_product_negative = -1.0  # Low similarity (learned)

prob_positive = sigmoid(dot_product_positive)
prob_negative = sigmoid(dot_product_negative)

Out[7]:

Console

Sigmoid Probabilities for Word Pairs:
--------------------------------------------------
Positive pair dot product: 2.5
  P(real pair) = sigma(2.5) = 0.9241

Negative pair dot product: -1.0
  P(real pair) = sigma(-1.0) = 0.2689

A well-trained model assigns high probability to
observed pairs and low probability to fake pairs.

The positive pair with dot product 2.5 receives probability 0.92, indicating high confidence that this is a observed context pair. The negative pair with dot product -1.0 receives only 0.27 probability, correctly reflecting low confidence.

Out[8]:

Visualization

S-shaped sigmoid curve showing probability output versus dot product, with positive and negative example points marked. — The sigmoid function maps dot products to probabilities. Large positive dot products (similar embeddings) yield probabilities near 1, indicating the model believes the pair comes from the corpus. Large negative dot products (dissimilar embeddings) yield probabilities near 0. The positive and negative example pairs from our computation are marked, showing that a trained model separates them decisively.