Attention Masking › Summary

Attention masking controls which positions can attend to which others, enabling autoregressive generation, efficient batch processing, and scalable attention over long sequences. The mechanism is remarkably simple: add large negative values to the scores of positions that should be blocked, and let softmax do the rest. The complexity lies not in the mechanism but in understanding when and why each type of mask is needed.

The key concepts from this chapter are:

• Additive masking: Adding large negative values before softmax drives attention weights to near-zero, effectively blocking those positions. No special logic is required in the attention function itself.
• Padding masks: Prevent attention to padding tokens when batching sequences of different lengths. Real tokens should not be influenced by meaningless padding values, and allowing such influence would corrupt representations and waste model capacity.
• Causal masks: Block attention to future positions, enforcing left-to-right information flow for autoregressive language models. This enables parallel training on full sequences while maintaining the correct sequential context constraint, collapsing $n$ sequential predictions into a single parallel forward pass.
• Mask combination: Multiple masks combine by addition. Any position blocked by any mask receives near-zero attention, implementing logical OR over the block conditions.
• Broadcasting: Masks can have shapes like (1, 1, seq_len, seq_len) for global patterns or (batch, 1, 1, seq_len) for per-sequence patterns, with broadcasting handling dimension expansion automatically and without extra memory allocation.
• Custom patterns: Local and strided attention, along with sparse patterns, reduce complexity for long sequences by limiting which positions can interact. These patterns trade some expressiveness for dramatic efficiency gains that make long-context modeling practical.
• Architectural choice: The choice of mask shapes what the model can learn. Causal masking suits generative models; bidirectional (no causal mask) attention suits discriminative models. This decision propagates through all downstream fine-tuning and deployment choices.

In the next chapter, we'll explore multi-head attention, which runs multiple attention operations in parallel with different learned projections. This allows the model to attend to information from different representation subspaces at different positions simultaneously, a capability that masking makes possible to apply selectively and efficiently.