Multilayer Perceptrons › MLP for Regression

Michael BrenndoerferApril 29, 20253 min read
Language AI HandbookAll chapters
Part 10 of 67Neural Network Foundations
Chapter 63Multilayer Perceptrons › MLP for Regression
Reading section 8 of 13 · MLP for Regression · 3 minSection 8 of 13 · 3 min

Section 8 of 13 · 3 min read

Multilayer Perceptrons › MLP for RegressionSection view · Sections in this chapter8 / 13 ↓
Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Regression tasks require predicting continuous values rather than class labels. The key differences from classification are:

  • No activation function on the output layer (identity function, allowing unbounded predictions)
  • Mean squared error (MSE) or mean absolute error (MAE) as the loss function
  • Output layer has one neuron per target variable
PythonIn [30]
CollapseExpand
# Create a regression dataset: sin(x) + 0.1 * x^2 with Gaussian noise
np.random.seed(42)
X_reg = np.random.uniform(-3, 3, (500, 1))
y_reg = np.sin(X_reg) + 0.1 * X_reg**2 + np.random.normal(0, 0.15, X_reg.shape)

X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(
    X_reg, y_reg, test_size=0.2, random_state=42
)

X_train_reg_t = torch.FloatTensor(X_train_reg)
y_train_reg_t = torch.FloatTensor(y_train_reg)
X_test_reg_t = torch.FloatTensor(X_test_reg)
y_test_reg_t = torch.FloatTensor(y_test_reg)
PythonIn [31]
CollapseExpand
class RegressionMLP(nn.Module):
    def __init__(self, input_size, hidden_sizes):
        super().__init__()

        layers = []
        prev_size = input_size

        for hidden_size in hidden_sizes:
            layers.append(nn.Linear(prev_size, hidden_size))
            layers.append(nn.ReLU())
            prev_size = hidden_size

        # Output layer with no activation for regression
        layers.append(nn.Linear(prev_size, 1))
        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return self.network(x)


model_reg = RegressionMLP(input_size=1, hidden_sizes=[32, 16])
criterion_reg = nn.MSELoss()
optimizer_reg = optim.Adam(model_reg.parameters(), lr=0.01)

train_losses_reg = []
for epoch in range(300):
    model_reg.train()
    y_pred_reg = model_reg(X_train_reg_t)
    loss = criterion_reg(y_pred_reg, y_train_reg_t)

    optimizer_reg.zero_grad()
    loss.backward()
    optimizer_reg.step()

    train_losses_reg.append(loss.item())
VisualizationsOut [32]
CollapseExpand
Scatter plot of noisy training data with true function (dashed) and MLP prediction (solid) overlaid.
MLP function approximation on a noisy sine-plus-quadratic dataset. The blue curve shows MLP predictions, the dashed orange line shows the true underlying function. The MLP recovers the non-linear shape despite noise.
Line plot of MSE training loss on logarithmic y-axis decreasing over 300 epochs.
Training loss (MSE) on a logarithmic scale over 300 epochs. The loss drops by roughly two orders of magnitude, indicating the network successfully learns the underlying function.
ConsoleOut [33]
CollapseExpand
Regression Results:
  Test MSE:  0.0245
  Test RMSE: 0.1564

Training Progress:
  Initial MSE: 0.8912
  Final MSE:   0.0220
  Improvement: 97.5%

The low test MSE and RMSE indicate the model fits the underlying function well. Unlike polynomial regression where you must choose the degree up front, the MLP automatically discovers the appropriate level of complexity through its hidden representations.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

1 comment

  1. Anonymous

    fantastic article

    1. Michael BrenndoerferMember

      Thank you very much for your comment!

Newsletter

Stay up to date

Get new articles, book updates, and notes on AI and technology.

Unsubscribe at any time. Privacy policy