Attention Masking › Key Parameters

When implementing attention masking, several parameters control the mask behavior:

• mask_value: The large negative value used for masked positions (typically $-10^9$ or -float('inf')). Using $-10^9$ rather than true infinity avoids numerical issues with some operations while still producing near-zero attention weights after softmax. If every position in a row is masked, using $-\infty$ produces NaN via $0/0$ in softmax, while $-10^9$ produces a uniform distribution that is mathematically wrong but numerically stable.

• window_size (local attention): Controls how many positions on each side a token can attend to. Smaller windows reduce computation but may miss important long-range dependencies. Common values range from 128 to 512 tokens in production systems.

• stride (strided attention): Determines the spacing between attended positions in sparse patterns. A stride of $k$ means attending to every $k$-th position, reducing complexity while maintaining some global connectivity.

• pad_token_id: The token ID used for padding in tokenized sequences. This must match the padding token used during tokenization to correctly identify positions to mask.