Multilayer Perceptrons › Key Parameters

When building MLPs in PyTorch, several parameters significantly impact model performance.

Architecture Parameters:

• hidden_sizes: List of neurons per hidden layer (e.g., [256, 128, 64]). Larger sizes increase capacity but also parameter count and risk of overfitting. Start with powers of 2 for computational efficiency.
• input_size: Number of input features, determined by your data.
• num_classes / output_size: Number of output neurons. Use 1 for binary classification or regression, $K$ for $K$-class classification.

Regularization Parameters:

• dropout_rate: Probability of zeroing neurons during training (typically 0.1-0.5). Higher values provide stronger regularization but may slow convergence. Use 0.2-0.3 as a starting point.
• weight_decay: L2 regularization strength in the optimizer (typically 1e-4 to 1e-2). Penalizes large weights to reduce overfitting.

Training Parameters:

• lr (learning rate): Step size for gradient updates (typically 1e-4 to 1e-1). Too high causes instability; too low causes slow convergence. Adam optimizer often works well with lr=0.001.
• epochs: Number of complete passes through the training data. Monitor validation loss to avoid overfitting.
• batch_size: Number of examples per gradient update (typically 32-256). Larger batches provide more stable gradients but require more memory.

Activation Choices:

• Hidden layers: ReLU is the default. Consider Leaky ReLU if many neurons die (output constant zero).
• Output layer: Sigmoid for binary classification, Softmax (via CrossEntropyLoss) for multiclass, Identity (none) for regression.

Loss Functions:

• nn.BCELoss(): Binary cross-entropy for binary classification with sigmoid output.
• nn.CrossEntropyLoss(): Combines softmax and negative log-likelihood for multiclass classification. Expects raw logits, not probabilities.
• nn.MSELoss(): Mean squared error for regression tasks.