MADALINE Neural Networks (1962) › The Mathematics of Adaptive Learning

The learning process in MADALINE centered on the LMS algorithm, which provided a mathematically principled yet computationally simple method for adjusting weights based on experience. Understanding how this algorithm worked reveals both the elegance of the approach and its fundamental limitations.

Each ADALINE unit computed its output as a weighted combination of its inputs. For a single ADALINE receiving inputs $x_1$ through $x_n$, the computation proceeded as follows. First, the unit formed a weighted sum:

Here, each input $x_j$ was multiplied by its corresponding weight $w_{ij}$, and these products were summed together along with a bias term $b_i$. This linear combination represented the unit's continuous response before any thresholding was applied. The subscript $i$ identified which ADALINE we were discussing, since MADALINE contained multiple such units operating in parallel.

During the learning phase, the system worked directly with these continuous values. This distinguished it from the perceptron. Rather than immediately converting the weighted sum to a binary decision, the learning algorithm used the full precision of the linear output. This allowed for smooth, stable weight updates that could make fine adjustments based on how far the output was from the desired value.

When the system needed to make an actual binary decision, it applied a threshold to convert the continuous output to a discrete choice:

These binary outputs $z_i$ from each ADALINE then fed into the fixed logic gates. For a simple AND gate combining the outputs of multiple units, the rule was straightforward: output 1 only if all inputs equaled 1, otherwise output -1. OR gates and more complex logical combinations could implement richer decision rules that reflected the structure of the problem being solved.

The heart of the learning process was the weight update rule. When the network made an error, meaning the final output did not match the desired result, the system needed to adjust the ADALINE weights to reduce that error. The LMS algorithm provided a remarkably simple prescription. For each weight, compute:

This formula embodied several intuitive principles. The term $(d - y_i)$ represented the error, the difference between what the ADALINE should have produced and the output it produced. The larger this error, the bigger the weight adjustment. The term $x_j$ meant that inputs which were more active had their weights adjusted more. If an input was zero or very small, its weight barely changed, which made sense since it was not contributing to the current output. The parameter $\alpha$, called the learning rate, controlled the size of weight updates. Engineers could adjust this to trade off between fast learning, which might be unstable, and slow but steady convergence.

This gradient descent approach had a satisfying mathematical interpretation. The LMS algorithm followed the negative gradient of the squared error function, taking steps downhill toward configurations of weights that minimized the mean squared difference between desired and actual outputs. Widrow and Hoff had created a learning rule that was both computationally simple enough to implement in 1960s hardware and theoretically grounded in optimization principles that would prove fundamental to machine learning.