Convolutional Neural Networks (1988) › The Mechanics of Convolution

To understand how CNNs work, we need to examine the core operation that gives them their name: convolution. This mathematical operation, borrowed from signal processing, provides the mechanism through which CNNs detect patterns in their input.

The convolutional layer applies small filters, also called kernels, to the input data. Each filter is a matrix of learned parameters, typically much smaller than the input itself. A common size is 3×3 or 5×5 elements. The filter slides across the input, one position at a time, computing at each position the sum of element-wise multiplications between the filter and the input region it currently covers. This process generates a feature map that indicates where the filter's pattern appears in the input.

Mathematically, for a position (i,j) in the output, we compute:

$$\text{Output}(i,j) = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} \text{Input}(i+m, j+n) \cdot \text{Filter}(m,n)$$

where k represents the size of the filter. This formula captures the essence of convolution: at each output position, we compute a weighted sum of input values in a local neighborhood, with the weights determined by the filter.

To build intuition, consider a simple 3×3 filter designed to detect vertical edges:

$$\begin{bmatrix} 1 & 0 & -1 \\ 1 & 0 & -1 \\ 1 & 0 & -1 \end{bmatrix}$$

This filter responds strongly when the left side of its receptive field contains bright pixels (multiplied by 1) and the right side contains dark pixels (multiplied by -1), producing a large positive response. A vertical edge, where brightness changes sharply from left to right, generates exactly this pattern. Conversely, uniform regions produce responses near zero, since the positive contributions from bright pixels on the left are canceled by the negative contributions from equally bright pixels on the right. The filter has learned to act as an edge detector, though in practice, CNNs learn filter values through training rather than having them hand-designed.

Each convolutional layer typically contains many filters, perhaps dozens or hundreds, each learning to detect different patterns. The first layer might learn filters that respond to edges at various orientations, corners, color contrasts, or simple textures. As we progress through the network, later layers combine responses from earlier layers to detect more complex patterns. A middle layer might have filters that respond to eyes, wheels, or windows by combining edge and texture detections from earlier layers. Deep layers might detect entire faces, cars, or buildings by combining the responses of middle-layer filters. This hierarchical composition of features is what allows CNNs to learn rich representations from raw pixel data.