Convolutional Neural Networks (1988) › Building Hierarchies of Understanding

The true power of CNNs emerges from how the layers work together to build increasingly sophisticated representations. This hierarchical organization mirrors how neuroscientists believe the visual cortex processes information, with simple features detected early and progressively combined into more complex representations.

In the first convolutional layer, which receives raw pixel values as input, the filters typically learn to detect basic visual primitives. Edge detectors emerge naturally, responding to transitions between light and dark regions. Some filters might detect edges oriented vertically, others horizontally, still others at diagonal angles. Beyond edges, first-layer filters often learn to detect simple color patterns, basic textures like grids or dots, and corners where edges meet. These are the building blocks from which all higher-level understanding is constructed.

The second layer operates on the feature maps produced by the first layer, meaning it processes patterns of edges and textures rather than raw pixels. At this level, filters learn to recognize combinations of the simple features detected earlier. A filter might respond to a specific configuration of edges that forms an arc or circle, or to a particular texture pattern created by a arrangement of oriented edges. The network is beginning to detect shapes and more complex textures, though still at a relatively local scale.

As we progress to deeper layers, the hierarchy continues. Middle layers might develop filters that respond to object parts: the wheel of a car (combining circular edges with specific texture patterns), the eye of a face (a particular arrangement of edges, colors, and shapes), or the petal of a flower (a specific combination of color patterns and curved edges). These part detectors are compositional, built from combinations of the simpler features detected in earlier layers.

The deepest layers combine these part detectors to recognize complete objects and even abstract concepts. A filter might respond specifically to faces by combining responses from eye detectors, nose detectors, and mouth detectors in the appropriate spatial configuration. Another might detect cars by looking for wheels, windows, and headlights in the right arrangement. The network has learned to build semantic understanding from the raw input, creating internal representations that capture meaningful concepts.

This hierarchical organization changes how the network learns. The network doesn't need to be told what features to look for, it discovers them through training. The features it learns are optimized for the specific task at hand, often surprising researchers with their sophistication. The hierarchical structure also allows knowledge to be reused: the same edge detectors useful for recognizing faces are equally valuable for recognizing cars, buildings, or animals. This sharing of learned features across different types of objects makes CNNs remarkably efficient learners.