Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Dropout › DropConnect

Michael BrenndoerferApril 27, 20253 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 72Dropout › DropConnect

Reading section 11 of 19 · DropConnect · 3 minSection 11 of 19 · 3 min

Section 11 of 19 · 3 min read

Dropout › DropConnectOne section · Sections in this chapter11 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

DropConnect is a generalization of dropout that extends the randomization from neuron outputs to individual weight connections. Instead of zeroing out a neuron's output (which zeroes its contribution to all downstream neurons), DropConnect randomly sets individual weights in the weight matrix to zero during training.

For a single linear layer in a standard network, the computation is:

h = \sigma(Wx + b)

With DropConnect, a binary mask is applied to the weight matrix:

h = \sigma((W \odot M)x + b)

where:

$W$ : the weight matrix of shape (output_dim, input_dim)
$M$ : a binary mask matrix of the same shape as $W$ , where each entry is 1 with probability $(1-p)$
$\odot$ : element-wise multiplication
$\sigma$ : the activation function

The key difference from dropout: dropout creates a single mask vector of length equal to the number of neurons, then applies it to the neuron's output (affecting all connections from that neuron to the next layer). DropConnect creates a mask matrix with one element per weight, allowing finer-grained randomization.

Dropout vs DropConnect

Dropout zeroes an entire neuron's output, which zeros all of its connections downstream. DropConnect zeroes individual weights, allowing some connections from a neuron to survive while others are dropped. DropConnect can be seen as applying a finer-grained mask than dropout.

In practice, DropConnect often performs similarly to dropout and is less commonly used because it's computationally more expensive (requires materializing the masked weight matrix). It was most competitive on small benchmark datasets. For large-scale training, the additional complexity is rarely justified.

In[20]:

Code

import torch
import torch.nn as nn
import torch.nn.functional as F


class DropConnectLinear(nn.Module):
    """
    Linear layer with DropConnect regularization.
    Randomly zeroes individual weight connections during training.
    """

    def __init__(self, in_features, out_features, p_drop=0.5):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.p_drop = p_drop
        self.weight = nn.Parameter(torch.randn(out_features, in_features) * 0.1)
        self.bias = nn.Parameter(torch.zeros(out_features))

    def forward(self, x):
        if self.training and self.p_drop > 0:
            keep_prob = 1.0 - self.p_drop
            mask = torch.bernoulli(torch.full_like(self.weight, keep_prob))
            masked_weight = (self.weight * mask) / keep_prob
        else:
            masked_weight = self.weight
        return F.linear(x, masked_weight, self.bias)

Out[21]:

Console

DropConnect drops individual weights: 50 total weights in this layer
Standard dropout drops neurons: 5 output neurons

DropConnect provides finer-grained regularization but is more expensive.
Dropout masks: 10 neurons -> affects 50 weight contributions
DropConnect masks: 50 individual weights independently