MADALINE Neural Networks (1962) › The ADALINE Foundation

To understand MADALINE, we must first examine its building block: the ADALINE, or ADAptive LINear Element. Widrow and Hoff had developed ADALINE slightly earlier, and it represented a subtle but important departure from the perceptron's approach to learning. Where Rosenblatt's perceptron used a hard threshold function that produced abrupt binary outputs, ADALINE maintained a continuous linear output during the learning process. This seemingly small change transformed the system's learning behavior.

The perceptron's hard threshold created a difficult optimization problem. When the weighted sum of inputs fell just on one side of the threshold, a tiny change in weights could flip the output completely, causing the learning algorithm to oscillate and struggle to converge. ADALINE avoided this problem by keeping the activation continuous during training. The unit still produced binary outputs for final decisions, but during learning, it worked with the smooth linear combination of its inputs. This smoothness meant that small changes in weights produced small changes in outputs, making the learning process far more stable and predictable.

The learning algorithm that Widrow and Hoff developed for ADALINE, known as the Least Mean Squares or LMS algorithm, exemplified the same pragmatic engineering philosophy that would characterize MADALINE. Rather than seeking mathematically optimal solutions that might be computationally intractable, LMS provided a simple, incremental learning rule that could be implemented efficiently in hardware. At each step, the algorithm computed the error between the desired output and the actual linear output, then adjusted the weights proportionally to reduce this error. This gradient descent approach would prove foundational to many later learning algorithms, including the backpropagation method that would eventually enable deep neural networks.