Multilayer Perceptrons and Backpropagation › Implementation in NumPy and Scikit-learn

The table compares both models on one held-out split. The full-batch training objective decreased during the displayed run, and the small relative gradient-check error shows agreement between the analytic and numerical derivatives on the checked inputs. Neither observation proves global convergence or better generalization. The decision-boundary figures below show the shape learned by each model; that graphical property is distinct from its test log loss.

The executable example uses fixed hyperparameters and does not implement validation-based selection. Its test data support the final metrics and illustrative plots. When tuning a model, reserve validation data from the training partition or use cross-validation, then evaluate the selected procedure on untouched test data.

The hyperparameters are not independent knobs; they trade off capacity, optimization stability, and generalization. The main ones are:

• n_hidden: the width of the hidden layer (default: 16). A larger value gives the network more capacity to represent complicated functions, but it also increases the number of parameters and can increase overfitting risk when the added capacity is not controlled by data and regularization. Start around 8 to 32 for low-dimensional tabular data.
• lr: the learning rate (default: 0.08). If it is too large, the loss can oscillate or diverge; if it is too small, training takes many epochs. Values around 0.01 to 0.1 are reasonable starting points for standardized inputs.
• batch_size: the maximum number of examples in each mini-batch gradient estimate (default: 64); the final batch may be smaller. At fixed parameters, smaller batches can give noisier gradient estimates. That randomness can sometimes move training out of a narrow basin, where the loss rises quickly around a local minimum, but it can also keep the objective from settling. Values around 32 to 128 are starting candidates for this small standardized example; choose among them with validation under a comparable training budget.
• epochs: the number of passes over the training data (default: 400). Choose this with early stopping on a validation split.
• lam: the L2 regularization strength (default: 1e-3). It shrinks the weights toward zero and can improve generalization, especially when the network is wide relative to the amount of training data. Values around 1e-4 to 1e-2 are reasonable starting ranges.
• seed: the seed for the random number generator (default: 0). Set it to an integer to ensure consistent initialization and mini-batch shuffling across runs.

The initialization scale is controlled inside init_params. We use a standard deviation proportional to $1/\sqrt{d}$ for the first layer and $1/\sqrt{h}$ for the output layer, which keeps the initial preactivations from growing too quickly with the number of inputs.

The implementation is deliberately small, and each function covers one conceptual step, described in its surrounding prose. The plain gradient descent loop in train_mlp makes the mechanics visible; an extended training loop may use an adaptive optimizer and validation-based early stopping. Early stopping can be added to either plain gradient descent or an adaptive optimizer.

At fixed parameters, averaging a uniformly sampled batch of 64 per-example gradients has no greater sampling variance than using one sampled example, and has lower variance in directions where the per-example gradients vary. During actual training the parameters change after each batch, so this fixed-parameter comparison does not guarantee a smoother or better optimization trajectory. For $N$ training examples and a positive integer maximum batch size $B$, this loop makes $\lceil N/B\rceil$ updates per epoch. The ceiling rounds $N/B$ upward to an integer. Reducing $B$ cannot lower that count, but nearby batch sizes can give the same count. Mini-batch gradient noise can also keep the training objective from settling. Compare training and validation loss: decreasing training loss with rising validation loss suggests overfitting, while both remaining high suggests underfitting or an optimization problem.

L2 regularization adds a penalty proportional to squared weights. It discourages large weights, which can reduce sensitivity to individual training points. Dropout changes the forward pass during training and is outside this implementation. Early stopping selects training duration with validation data: retain the parameters from the best observed validation epoch and stop after a chosen period without improvement.

Hidden units with identical incoming weights, biases, and outgoing weights receive identical gradients under the stated updates, so they remain identical. Random initialization helps separate their starting points. Input scales can make different weight gradients very different in magnitude. A learning rate that is too high can make the loss explode; one that is too low can make progress hard to see. If a first-layer ReLU unit is negative on every training input, its data-loss gradient is zero. With plain gradient descent and no other parameter updates, such a unit cannot recover. Tanh and sigmoid can instead saturate at large preactivations, making their local derivatives $1-a^2$ or $p(1-p)$ small.