Skip-gram Model › Architecture: Two Embedding Matrices

The Skip-gram architecture is a shallow neural network with one hidden layer and no nonlinear activation function. Its apparent simplicity is misleading, because all the meaningful computation happens in the weight matrices, not in any deep stacking of transformations.

The network contains three components:

• Input layer: A one-hot vector of dimension $V$ (vocabulary size), with a 1 at the index of the center word and 0 everywhere else.
• Hidden layer: A weight matrix $\mathbf{W}$ of shape $V \times d$, where $d$ is the embedding dimension. Multiplying the one-hot input by $\mathbf{W}$ selects the $d$-dimensional row corresponding to the center word.
• Output layer: A second weight matrix $\mathbf{W}'$ of shape $d \times V$ that projects the hidden vector back to vocabulary size, producing a score for every word.

After training, we discard the output matrix and use $\mathbf{W}$ as our embedding lookup table. Each row of $\mathbf{W}$ is the embedding for the corresponding word.

The absence of a nonlinear activation function between the input embedding lookup and the output scoring is intentional. Adding nonlinearities would make the model harder to optimize and would not obviously improve the quality of the learned representations. The linear structure means the model is essentially learning a bilinear scoring function between center word embeddings and context word embeddings, and the geometry of that scoring function is what creates the useful structure in the embedding space.

One-hot encoding gives each word a unique, orthogonal representation. Word $w$ with vocabulary index $i$ is represented as a vector $\mathbf{x}$ of length $V$ where $x_i = 1$ and all other entries are 0. This is not a meaningful representation on its own; its purpose is purely to select a row from the embedding matrix.

Multiplying a one-hot vector $\mathbf{x}$ by the embedding matrix $\mathbf{W}$ is equivalent to selecting row $i$:

$$\mathbf{h} = \mathbf{W}^T \mathbf{x} = \mathbf{v}_{w_I}$$

where:

• $\mathbf{h}$: the hidden layer vector of dimension $d$, which is the center word embedding
• $\mathbf{W}$: the input embedding matrix of shape $V \times d$
• $\mathbf{x}$: the one-hot input vector with a 1 at position $i$
• $\mathbf{v}_{w_I}$: the $i$-th row of $\mathbf{W}$, the embedding for input word $w_I$

This selection operation is the key to Skip-gram's efficiency. We never multiply a full one-hot vector through a large matrix in practice. We simply look up a row. Because the one-hot vector is all zeros except at one position, the matrix multiplication collapses to a single row retrieval. Modern implementations store word embeddings in a dedicated Embedding lookup table, which is exactly this: a parameterized table where each row is a learnable vector, and integer indexing is used to retrieve the relevant row.

The output matrix $\mathbf{W}'$ of shape $d \times V$ computes a score for each vocabulary word. For context word $w_O$ with vocabulary index $j$, the score is:

$$u_j = \mathbf{v}'_{w_O} \cdot \mathbf{h}$$

where:

• $u_j$: the unnormalized score (logit) for vocabulary word $j$ being a context word of $w_I$
• $\mathbf{v}'_{w_O}$: the $j$-th column of $\mathbf{W}'$, the context embedding of word $w_O$
• $\mathbf{h}$: the center word embedding from the hidden layer

So every word has two vector representations: an input embedding (a row of $\mathbf{W}$) used when the word is the center, and a context embedding (a column of $\mathbf{W}'$) used when it appears in someone else's context window. This dual representation is a fundamental design feature of Word2Vec. After training, you typically use the input embeddings ($\mathbf{W}$) as the final word vectors, though some practitioners average both matrices, or use the context embeddings for specific downstream tasks.

The use of separate input and context matrices deserves explanation, because it might seem redundant at first. Why should the embedding for "fox" as a center word differ from its embedding as a context word?

The reason is flexibility and symmetry breaking. If we used the same matrix for both roles, the model would be constrained to treat each word as both predictor and predicted in a symmetric way. With separate matrices, the model can learn that "fox" as a center word needs to be good at pointing toward animal-related words in context space, while "fox" as a context word needs to be pointed to by sentences that often contain fox-relevant contexts. These are related but not identical requirements, and separate matrices allow the model to satisfy them independently.

This also has an interesting geometric consequence. If you compute the dot product $\mathbf{v}_{w_I} \cdot \mathbf{v}'_{w_O}$, you are measuring how well word $w_I$ as center predicts word $w_O$ as context, which is not the same as asking whether $w_O$ as center predicts $w_I$ as context (that would be $\mathbf{v}_{w_O} \cdot \mathbf{v}'_{w_I}$). The model naturally handles asymmetric co-occurrence statistics this way.