Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Weight Initialization › Orthogonal Initialization

Michael BrenndoerferApril 25, 20253 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 70Weight Initialization › Orthogonal Initialization

Reading section 9 of 19 · Orthogonal Initialization · 3 minSection 9 of 19 · 3 min

Section 9 of 19 · 3 min read

Weight Initialization › Orthogonal InitializationOne section · Sections in this chapter9 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Xavier and He initialization focus on setting the right variance. Orthogonal initialization takes a different approach: it constructs weight matrices whose columns are perpendicular unit vectors, a property called orthogonality.

An orthogonal matrix $W$ satisfies $W^T W = I$ (the identity matrix). This means the transformation preserves the length of vectors: $\|Wx\|_2 = \|x\|_2$ for any input vector $x$ . In terms of variance, orthogonal initialization guarantees exact variance preservation during the forward pass, with no dependence on the input distribution or layer size.

Orthogonal Initialization

A weight initialization scheme that constructs square weight matrices to be orthogonal, satisfying $W^T W = I$ . Since orthogonal transformations preserve vector norms, this guarantees exact variance preservation during the forward pass and is particularly valuable for recurrent neural networks where the same matrix is applied repeatedly across time steps.

The construction uses singular value decomposition (SVD). Starting from a random matrix $A$ with entries drawn from $\mathcal{N}(0, 1)$ , we compute the SVD $A = U \Sigma V^T$ and use $U$ (or $V^T$ ) as the initial weight matrix. Both $U$ and $V$ are unitary matrices satisfying the orthogonality condition.

Why Orthogonality Matters for Gradient FlowLink Copied

Beyond variance preservation, orthogonal matrices have an important property: their eigenvalues all have magnitude exactly 1. For a general matrix $W$ , the eigenvalues can be larger or smaller than 1 in magnitude. When a matrix with eigenvalues much larger than 1 is applied repeatedly (as in a recurrent network processing a long sequence), the signal grows exponentially. When eigenvalues are much smaller than 1, the signal decays exponentially. This is the spectral radius perspective on exploding and vanishing gradients.

An orthogonal matrix, with eigenvalues of magnitude exactly 1, neither amplifies nor attenuates the signal regardless of how many times it is applied. This makes orthogonal initialization particularly well-suited for recurrent neural networks, where the same hidden-to-hidden weight matrix is multiplied at every time step. Even for very long sequences (hundreds or thousands of steps), an orthogonally initialized recurrent matrix maintains bounded signal propagation.

In[19]:

Code

def orthogonal_init(shape, gain=1.0):
    """
    Orthogonal initialization using SVD.

    Parameters:
    - shape: (fan_in, fan_out)
    - gain: scaling factor (use sqrt(2) for ReLU networks)
    """
    fan_in, fan_out = shape
    a = np.random.randn(fan_in, fan_out)
    U, _, Vt = np.linalg.svd(a, full_matrices=False)
    # Choose U or Vt depending on shape
    q = U if U.shape == (fan_in, fan_in) else Vt
    if q.shape != shape:
        q = q[:fan_in, :fan_out]
    return gain * q


def get_variance_after_layers(init_func, n_layers=20, layer_size=256):
    """Measure variance across layers in a linear network."""
    weights = [init_func((layer_size, layer_size)) for _ in range(n_layers)]
    x = np.random.randn(1000, layer_size)
    x = x / np.std(x)
    variances = [np.var(x)]
    for W in weights:
        x = x @ W
        variances.append(np.var(x))
    return variances

Out[20]:

Console

Variance preservation (20-layer linear network, no activation):
------------------------------------------------------------


Random normal (std=1.0):
  Initial: 1.0000, Final: 1.5142e+48

He normal:
  Initial: 1.0000, Final: 1.0017e+06

Orthogonal:
  Initial: 1.0000, Final: 1.0000e+00

Orthogonal initialization achieves near-perfect variance preservation in linear networks. This makes it particularly valuable for recurrent neural networks, where the hidden-to-hidden weight matrix is applied at every time step. We'll explore this in more depth when we cover RNNs.

For feedforward networks, the practical advantages of orthogonal initialization over He are less clear-cut, because activation functions break the orthogonality property immediately after the first layer. Xavier and He initialization work well for typical feedforward architectures, while orthogonal initialization is the preferred choice for RNN hidden-to-hidden matrices.