Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multilayer Perceptrons › Representational Capacity and the Universal Approximation Theorem

Michael BrenndoerferApril 29, 20254 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 63Multilayer Perceptrons › Representational Capacity and the Universal Approximation Theorem

Reading section 6 of 13 · Representational Capacity and the Universal Approximation Theorem · 4 minSection 6 of 13 · 4 min

Section 6 of 13 · 4 min read

Multilayer Perceptrons › Representational Capacity and the Universal Approximation TheoremOne section · Sections in this chapter6 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of the most remarkable properties of MLPs is their ability to approximate any continuous function. The Universal Approximation Theorem formalizes this power.

Universal Approximation Theorem

A feedforward network with a single hidden layer containing a finite number of neurons, and a non-polynomial activation function, can approximate any continuous function on a compact subset of $\mathbb{R}^n$ to arbitrary accuracy. This holds given appropriate weights, but does not guarantee that gradient descent will find such weights.

The theorem tells us that MLPs are expressive enough to represent complex functions. However, it says nothing about:

How many neurons are needed (potentially exponentially many for a single hidden layer)
Whether training will find good weights
How well the network will generalize to new data

In practice, deeper networks often work better than wider networks for the same total number of parameters. Depth enables hierarchical feature learning, where early layers detect simple patterns and later layers combine them into complex concepts.

Depth vs. Width TradeoffLink Copied

Depth and width trade off representational capacity in different ways. A shallow but wide network might require exponentially more neurons to represent the same function as a moderately deep network. The intuition is similar to building with modular components: a few large building blocks can cover area, but many small components arranged hierarchically create more complex structures with fewer total pieces.

Consider three architectures with similar parameter budgets but different depth/width tradeoffs:

In[16]:

Code

# Demonstrate function approximation with varying network sizes
def create_mlp(layer_sizes):
    """Create an MLP with given layer sizes using Xavier initialization."""
    np.random.seed(42)
    w_list = []
    b_list = []
    for l in range(1, len(layer_sizes)):
        scale = np.sqrt(2.0 / (layer_sizes[l - 1] + layer_sizes[l]))
        W = np.random.randn(layer_sizes[l], layer_sizes[l - 1]) * scale
        b = np.zeros((layer_sizes[l], 1))
        w_list.append(W)
        b_list.append(b)
    return w_list, b_list


shallow_mlp = create_mlp([128, 64, 1])  # 1 hidden layer, 64 neurons
medium_mlp = create_mlp([128, 32, 32, 1])  # 2 hidden layers, 32 neurons each
deep_mlp = create_mlp([128, 16, 16, 16, 1])  # 3 hidden layers, 16 neurons each


def count_params(w_list, b_list):
    return sum(W.size for W in w_list) + sum(b.size for b in b_list)

Out[17]:

Console

Network architectures and parameter counts:
  Shallow (128-64-1):        8,321 parameters  (1 hidden layer)
  Medium  (128-32-32-1):     5,217 parameters  (2 hidden layers)
  Deep    (128-16-16-16-1):  2,625 parameters  (3 hidden layers)

The shallow network has the most parameters despite having only one hidden layer. The deeper networks distribute their capacity across more layers with fewer neurons each. Let's visualize this tradeoff:

Out[18]:

Visualization

Stacked bar chart comparing parameter counts in shallow, medium, and deep networks. — Parameter distribution across three network architectures with different depth/width tradeoffs. Despite having more layers, the deeper network has fewer total parameters because each hidden layer is narrower. The first connection from the 128-dimensional input dominates parameter count in all cases.

This tradeoff matters in practice. Depth enables compositionality: the network can learn that "a phrase that contains a question word followed by a verb" is a pattern, without needing to enumerate every possible such combination.