Global Tokens › Learned Global Tokens
Language AI HandbookAll chapters Global Tokens › Learned Global TokensOne section · Sections in this chapter5 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
CLS and task-specific tokens are fixed by design, meaning a human engineer must decide which positions serve as global hubs before training begins. An alternative approach introduces learnable global tokens that the model discovers during training. These tokens do not correspond to any input words but serve purely as memory and communication buffers.
This shift from designated to learned global tokens represents a deeper change in perspective. With designated tokens, we are imposing prior knowledge about the task structure onto the architecture: we believe the question tokens are globally important, so we encode that belief in the mask. With learned global tokens, we step back and say: we do not know exactly what global information the task needs, so let the model figure it out. The learned tokens become a trainable bottleneck through which global information flows, and the model optimizes how to use them to minimize task loss.
Perceiver and Perceiver IO use this idea extensively. They introduce a small set of latent tokens, often 256 to 512, that cross-attend to a much longer input sequence. The latent tokens then self-attend among themselves efficiently, and finally cross-attend back to produce outputs. This three-stage process, cross-attend in, self-attend internally, cross-attend out, allows the model to process inputs of arbitrary length while keeping the expensive self-attention computation confined to the small latent space.
The elegance of learned global tokens is their generality. The same mechanism works whether the input is text, images, audio, or structured data. The latent tokens do not know or care about the modality of the input. They simply learn to extract whatever information is relevant for the task from whatever input is presented. This is why Perceiver IO succeeded at processing multimodal inputs: the latent tokens served as a modality-agnostic interface between the raw input stream and the task-specific output.
Think of learned global tokens as empty vessels at initialization. At the start of training, their embeddings are random. This provides no useful information. As training proceeds, the gradient signal shapes them to carry exactly the information that other tokens need to perform the task. By the end of training, each latent token has a specialized role in the model's information routing scheme, even though that role was never explicitly programmed.
def create_latent_global_mask(input_len, num_latents, window_size):
"""
Create attention mask with learned latent global tokens.
Latent tokens attend globally; input tokens attend locally + to latents.
"""
total_len = num_latents + input_len
mask = np.zeros((total_len, total_len), dtype=bool)
# Latent tokens (positions 0 to num_latents-1) have global attention
for i in range(num_latents):
mask[i, :] = True
mask[:, i] = True
# Input tokens use sliding window among themselves
for i in range(num_latents, total_len):
input_pos = i - num_latents
# Local window in input space
start = num_latents + max(0, input_pos - window_size // 2)
end = num_latents + min(input_len, input_pos + window_size // 2 + 1)
mask[i, start:end] = True
return mask
# Example: 4 latent tokens, 16 input tokens
num_latents = 4
input_len = 16
window_size = 4
latent_mask = create_latent_global_mask(input_len, num_latents, window_size)
The advantage of learned latents is flexibility. The model can discover what information to store in global memory rather than being constrained to predefined positions. The latent tokens act as a compressed representation of the full sequence, enabling efficient long-range communication.
In practice, learned global tokens tend to specialize organically. Probing studies on Perceiver models have found that different latent tokens attend preferentially to different parts of the input or different semantic categories of information, even though no such specialization was explicitly programmed. One latent might concentrate its attention on entities, another on relational phrases, another on discourse markers. This emergent specialization is a sign that the model is efficiently using its global token budget, allocating different aspects of sequence-level understanding to different latent positions.