Feed-Forward Networks › Position Independence
Language AI HandbookAll chapters Feed-Forward Networks › Position IndependenceOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A important property of the FFN is that it processes each position independently. Unlike attention, where every position can influence every other position through the query-key-value mechanism, the FFN applies an identical transformation to each position in isolation. This has practical consequences for both computation and interpretation, and it reflects a deep design decision about the division of responsibilities in the transformer architecture.
When we say the FFN is position-independent, we mean that the output at position depends only on the input at position . The FFN does not know or care about adjacent tokens. It does not look at the whole sequence. It simply applies the same transformation to whatever vector it receives, and that vector could have come from any position in the sequence.
This might seem like a limitation. After all, language is deeply contextual. The meaning of "bank" depends on whether "river" or "money" appeared nearby. But here is the key insight: by the time the FFN sees the representation at position , that representation has already been processed by the self-attention sublayer in the same block. Attention has already incorporated contextual information. The FFN is operating on a context-enriched representation, not on a raw token embedding. Its job is to transform that already-contextualized representation further, not to gather more context.
Think of the transformer block as a two-phase processor: attention gathers context (inter-position), and the FFN processes the gathered context (intra-position). The FFN can focus entirely on the "what to do with this information" question because attention has already handled the "what context is relevant" question.
Let's verify this independence empirically:
# Demonstrate position independence
seq_len = 5
X = np.random.randn(seq_len, d_model)
# Process all positions at once (batch processing)
Y_batch = ffn(X, W1, b1, W2, b2)
# Process each position individually
Y_individual = np.zeros_like(X)
for i in range(seq_len):
Y_individual[i] = ffn(X[i], W1, b1, W2, b2)
# Check they're identical
difference = np.abs(Y_batch - Y_individual).max()Position independence verification: Maximum difference between batch and individual processing: 3.77e-15 Outputs are identical: True
The batch and individual processing produce identical results, up to floating-point precision. This confirms that positions don't interact within the FFN. Changing the token at position 3 has absolutely no effect on the FFN output at position 1, even if they share the same FFN weights. The information barrier between positions is absolute.
Position independence means the FFN can be computed in parallel across all positions. On a GPU, this is extremely efficient: instead of processing tokens sequentially, we process the entire sequence simultaneously as a batch matrix multiplication. The FFN is embarrassingly parallel, meaning no position needs to wait for any other position's result. For a sequence of 1024 tokens, we can compute all 1024 FFN outputs simultaneously, limited only by the size of the weight matrices and the amount of memory available.
Position independence also clarifies the division of labor in a transformer block. Attention handles inter-position communication: it routes information between tokens, allowing the model to build representations that depend on context. The FFN handles intra-position transformation: it transforms each position's representation using the same learned function, adding nonlinearity and processing capacity to what would otherwise be a purely linear attention mechanism. This division of labor is one of the transformer's most elegant architectural properties.
