Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multilayer Perceptrons › MLP for Regression

Michael BrenndoerferApril 29, 20253 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 63Multilayer Perceptrons › MLP for Regression

Reading section 8 of 13 · MLP for Regression · 3 minSection 8 of 13 · 3 min

Section 8 of 13 · 3 min read

Multilayer Perceptrons › MLP for RegressionOne section · Sections in this chapter8 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Regression tasks require predicting continuous values rather than class labels. The key differences from classification are:

No activation function on the output layer (identity function, allowing unbounded predictions)
Mean squared error (MSE) or mean absolute error (MAE) as the loss function
Output layer has one neuron per target variable

In[30]:

Code

# Create a regression dataset: sin(x) + 0.1 * x^2 with Gaussian noise
np.random.seed(42)
X_reg = np.random.uniform(-3, 3, (500, 1))
y_reg = np.sin(X_reg) + 0.1 * X_reg**2 + np.random.normal(0, 0.15, X_reg.shape)

X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(
    X_reg, y_reg, test_size=0.2, random_state=42
)

X_train_reg_t = torch.FloatTensor(X_train_reg)
y_train_reg_t = torch.FloatTensor(y_train_reg)
X_test_reg_t = torch.FloatTensor(X_test_reg)
y_test_reg_t = torch.FloatTensor(y_test_reg)

In[31]:

Code

class RegressionMLP(nn.Module):
    def __init__(self, input_size, hidden_sizes):
        super().__init__()

        layers = []
        prev_size = input_size

        for hidden_size in hidden_sizes:
            layers.append(nn.Linear(prev_size, hidden_size))
            layers.append(nn.ReLU())
            prev_size = hidden_size

        # Output layer with no activation for regression
        layers.append(nn.Linear(prev_size, 1))
        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return self.network(x)


model_reg = RegressionMLP(input_size=1, hidden_sizes=[32, 16])
criterion_reg = nn.MSELoss()
optimizer_reg = optim.Adam(model_reg.parameters(), lr=0.01)

train_losses_reg = []
for epoch in range(300):
    model_reg.train()
    y_pred_reg = model_reg(X_train_reg_t)
    loss = criterion_reg(y_pred_reg, y_train_reg_t)

    optimizer_reg.zero_grad()
    loss.backward()
    optimizer_reg.step()

    train_losses_reg.append(loss.item())

Out[32]:

Visualization

Scatter plot of noisy training data with true function (dashed) and MLP prediction (solid) overlaid. — MLP function approximation on a noisy sine-plus-quadratic dataset. The blue curve shows MLP predictions, the dashed orange line shows the true underlying function. The MLP recovers the non-linear shape despite noise.

Line plot of MSE training loss on logarithmic y-axis decreasing over 300 epochs. — Training loss (MSE) on a logarithmic scale over 300 epochs. The loss drops by roughly two orders of magnitude, indicating the network successfully learns the underlying function.

Out[33]:

Console

Regression Results:
  Test MSE:  0.0245
  Test RMSE: 0.1564

Training Progress:
  Initial MSE: 0.8912
  Final MSE:   0.0220
  Improvement: 97.5%

The low test MSE and RMSE indicate the model fits the underlying function well. Unlike polynomial regression where you must choose the degree up front, the MLP automatically discovers the appropriate level of complexity through its hidden representations.