Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multilayer Perceptrons › MLP for Classification

Michael BrenndoerferApril 29, 20259 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 63Multilayer Perceptrons › MLP for Classification

Reading section 7 of 13 · MLP for Classification · 9 minSection 7 of 13 · 9 min

Section 7 of 13 · 9 min read

Multilayer Perceptrons › MLP for ClassificationOne section · Sections in this chapter7 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Classification is one of the most common tasks for MLPs. The network takes features as input and produces probabilities for each class. For binary classification, we use a single output neuron with sigmoid activation. For multiclass, we use multiple output neurons with softmax activation.

Binary ClassificationLink Copied

In binary classification, the output $\hat{y} \in (0, 1)$ represents the probability that the input belongs to class 1. We train the network by minimizing the binary cross-entropy loss:

\mathcal{L} = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \right]

where:

$m$ : the number of training examples in the batch
$y^{(i)}$ : the true label for example $i$ (either 0 or 1)
$\hat{y}^{(i)}$ : the predicted probability that example $i$ belongs to class 1

When the true label is $y^{(i)} = 1$ , only the first term $-\log(\hat{y}^{(i)})$ is active, penalizing low predicted probabilities. When $y^{(i)} = 0$ , only the second term $-\log(1 - \hat{y}^{(i)})$ is active, penalizing high predicted probabilities. The loss approaches zero when predictions are confident and correct, and grows very large when predictions are confidently wrong.

Out[19]:

Visualization

Line plot showing binary cross-entropy loss curves for y=1 and y=0 cases across predicted probabilities. — Binary cross-entropy loss for different true labels. When y=1 (blue), the loss decreases as predicted probability increases, rewarding confident correct predictions. When y=0 (orange), the loss decreases as predicted probability decreases. The steep rise near incorrect confident predictions provides strong learning signals.

Let's build a complete binary classifier using PyTorch on the two moons dataset, a classic non-linearly separable benchmark:

In[20]:

Code

import torch
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Create a non-linearly separable dataset
X_moons, y_moons = make_moons(n_samples=1000, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X_moons, y_moons, test_size=0.2, random_state=42
)

# Standardize features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Convert to PyTorch tensors
X_train_t = torch.FloatTensor(X_train)
y_train_t = torch.FloatTensor(y_train).reshape(-1, 1)
X_test_t = torch.FloatTensor(X_test)
y_test_t = torch.FloatTensor(y_test).reshape(-1, 1)

In[21]:

Code

import torch.nn as nn
import torch.optim as optim


class BinaryClassifierMLP(nn.Module):
    def __init__(self, input_size, hidden_sizes, dropout_rate=0.2):
        super().__init__()

        layers = []
        prev_size = input_size

        for hidden_size in hidden_sizes:
            layers.append(nn.Linear(prev_size, hidden_size))
            layers.append(nn.ReLU())
            layers.append(nn.Dropout(dropout_rate))
            prev_size = hidden_size

        # Output layer with sigmoid for binary classification
        layers.append(nn.Linear(prev_size, 1))
        layers.append(nn.Sigmoid())

        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return self.network(x)


# Create model: 2 inputs -> 16 -> 8 -> 1 output
model = BinaryClassifierMLP(input_size=2, hidden_sizes=[16, 8])
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

In[22]:

Code

# Training loop
train_losses = []
test_accuracies = []

for epoch in range(200):
    model.train()
    y_pred = model(X_train_t)
    loss = criterion(y_pred, y_train_t)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    train_losses.append(loss.item())

    model.eval()
    with torch.no_grad():
        y_test_pred = model(X_test_t)
        accuracy = ((y_test_pred > 0.5) == y_test_t).float().mean().item()
        test_accuracies.append(accuracy)

Out[23]:

Console

Training Results:
  Final training loss: 0.2116
  Final test accuracy: 95.50%
  Initial loss:        0.6903
  Loss reduction:      69.3%

The model achieves strong test accuracy after 200 epochs. The significant reduction in loss indicates successful learning. Let's visualize the decision boundary:

Out[24]:

Visualization

Contour plot with a curved decision boundary separating two moon-shaped clusters of points. — Decision boundary learned by the MLP on the two moons dataset. The curved boundary successfully separates the two interleaved half-moon shapes, a task impossible for any linear classifier.

Out[25]:

Visualization

Dual-axis line plot showing training loss decreasing and test accuracy increasing over 200 epochs. — Training progress showing loss decrease and accuracy improvement over 200 epochs. Loss drops rapidly in the first 50 epochs as the network learns the basic structure, then fine-tunes more slowly.

The MLP learns a curved decision boundary that cleanly separates the two moon-shaped clusters. A linear classifier would fail entirely on this task, but the hidden layer converts the input into a representation where the classes become separable.

Multiclass ClassificationLink Copied

For problems with more than two classes, we use softmax activation in the output layer. Softmax converts a vector of raw scores (called logits) into a probability distribution. Given an output vector $\mathbf{z} = [z_1, z_2, \ldots, z_K]$ from the final layer, softmax computes the probability for each class:

\text{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

where:

$K$ : the total number of classes
$z_i$ : the logit (raw score) for class $i$
$e^{z_i}$ : the exponential of $z_i$ , which ensures all values become positive
$\sum_{j=1}^{K} e^{z_j}$ : the sum of exponentials across all classes, serving as a normalization constant

The exponential function amplifies differences between logits. If one class has a much higher score than others, it will dominate the probability distribution. The denominator ensures all probabilities sum to 1.

Out[26]:

Visualization

Bar chart showing raw logit values for three classes. — Raw logits (unnormalized scores) from the network's output layer, which can be any real number including negatives.

Bar chart showing softmax probabilities for three classes summing to 1.0. — After softmax, the scores become a valid probability distribution that sums to 1.0. Class A dominates because its logit (2.0) is largest.

In[27]:

Code

from sklearn.datasets import load_iris

# Load iris dataset (3 classes)
iris = load_iris()
X_iris, y_iris = iris.data, iris.target
X_train_iris, X_test_iris, y_train_iris, y_test_iris = train_test_split(
    X_iris, y_iris, test_size=0.2, random_state=42
)

scaler_iris = StandardScaler()
X_train_iris = scaler_iris.fit_transform(X_train_iris)
X_test_iris = scaler_iris.transform(X_test_iris)

X_train_iris_t = torch.FloatTensor(X_train_iris)
y_train_iris_t = torch.LongTensor(y_train_iris)
X_test_iris_t = torch.FloatTensor(X_test_iris)
y_test_iris_t = torch.LongTensor(y_test_iris)

In[28]:

Code

class MulticlassClassifierMLP(nn.Module):
    def __init__(self, input_size, hidden_sizes, num_classes):
        super().__init__()

        layers = []
        prev_size = input_size

        for hidden_size in hidden_sizes:
            layers.append(nn.Linear(prev_size, hidden_size))
            layers.append(nn.ReLU())
            layers.append(nn.Dropout(0.2))
            prev_size = hidden_size

        # Output layer: no softmax since CrossEntropyLoss includes it
        layers.append(nn.Linear(prev_size, num_classes))
        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return self.network(x)


model_mc = MulticlassClassifierMLP(
    input_size=4, hidden_sizes=[16, 8], num_classes=3
)
criterion_mc = nn.CrossEntropyLoss()
optimizer_mc = optim.Adam(model_mc.parameters(), lr=0.01)

for epoch in range(200):
    model_mc.train()
    logits = model_mc(X_train_iris_t)
    loss = criterion_mc(logits, y_train_iris_t)

    optimizer_mc.zero_grad()
    loss.backward()
    optimizer_mc.step()

Out[29]:

Console

Test Results:
  Accuracy: 100.00% (30/30 correct)

Sample predictions (first 5 test examples):
  Classes: [np.str_('setosa'), np.str_('versicolor'), np.str_('virginica')]

  [correct] True: versicolor   | Predicted: versicolor   | Probs: [0.001, 0.997, 0.002]
  [correct] True: setosa       | Predicted: setosa       | Probs: [1.000, 0.000, 0.000]
  [correct] True: virginica    | Predicted: virginica    | Probs: [0.000, 0.000, 1.000]
  [correct] True: versicolor   | Predicted: versicolor   | Probs: [0.002, 0.989, 0.009]
  [correct] True: versicolor   | Predicted: versicolor   | Probs: [0.001, 0.994, 0.005]

The multiclass model achieves high accuracy on the iris dataset, correctly classifying most test samples. Notice how the softmax outputs sum to 1.0 for each example, creating valid probability distributions. When the model is confident, one class dominates with probability close to 1.0 while others are near zero.