Multilayer Perceptrons › Architecture Design Guidelines
Language AI HandbookAll chapters Multilayer Perceptrons › Architecture Design GuidelinesOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Designing an MLP architecture involves choosing the number of layers, neurons per layer, activation functions, and regularization techniques. While there is no universal formula, several principles guide these decisions.
Depth vs. Width
Deeper networks can represent more complex hierarchical features. However, they are harder to train due to vanishing or exploding gradients. Wider networks have more parameters per layer but may struggle to learn compositional patterns.
As a starting point:
- For simple problems, 1-2 hidden layers often suffice
- For complex patterns, 3-5 hidden layers may be needed
- Very deep networks (10+ layers) typically require residual connections to avoid gradient issues
Layer Sizes
Common patterns for hidden layer sizes include:
- Funnel: Decreasing sizes (e.g., 256-128-64) that progressively compress information
- Constant: Same size throughout (e.g., 128-128-128) for uniform capacity
- Bottleneck: Narrow middle layer (e.g., 256-32-256) to force compressed representations
The input and output sizes are determined by your problem. Hidden sizes are hyperparameters to tune based on validation performance.
Activation Functions
ReLU is the default choice for hidden layers due to its simplicity and effectiveness. Alternatives help in specific situations:
- ReLU: Fast, simple, works well in most cases
- Leaky ReLU/PReLU: Addresses the dying ReLU problem where neurons output constant zero
- GELU: Smooth approximation, popular in transformers
- Tanh/Sigmoid: Rarely used in hidden layers now due to saturation
For output layers:
- Sigmoid: Binary classification (outputs probability in range (0, 1))
- Softmax: Multiclass classification (outputs probability distribution)
- Identity (none): Regression (outputs unbounded values)
Regularization
To prevent overfitting, we use regularization techniques:
- Dropout: Randomly zeros neurons during training (typical rates: 0.2-0.5)
- Weight decay: L2 penalty on weights (typical values: 1e-4 to 1e-2)
- Batch normalization: Normalizes layer inputs, also acts as a regularizer
class WellDesignedMLP(nn.Module):
"""An MLP following best practices for architecture design."""
def __init__(
self,
input_size,
num_classes,
hidden_sizes=[256, 128, 64],
dropout_rate=0.3,
use_batch_norm=True,
):
super().__init__()
layers = []
prev_size = input_size
for hidden_size in hidden_sizes:
layers.append(nn.Linear(prev_size, hidden_size))
if use_batch_norm:
layers.append(nn.BatchNorm1d(hidden_size))
layers.append(nn.ReLU())
layers.append(nn.Dropout(dropout_rate))
prev_size = hidden_size
layers.append(nn.Linear(prev_size, num_classes))
self.network = nn.Sequential(*layers)
def forward(self, x):
return self.network(x)Well-designed MLP architecture:
WellDesignedMLP(
(network): Sequential(
(0): Linear(in_features=100, out_features=256, bias=True)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.3, inplace=False)
(4): Linear(in_features=256, out_features=128, bias=True)
(5): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
(6): ReLU()
(7): Dropout(p=0.3, inplace=False)
(8): Linear(in_features=128, out_features=64, bias=True)
(9): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
(10): ReLU()
(11): Dropout(p=0.3, inplace=False)
(12): Linear(in_features=64, out_features=10, bias=True)
)
)
Parameter count:
Total parameters: 68,554
Trainable parameters: 68,554