Vision Encoders for VLMs › SigLIP: Rethinking Vision-Language Pretraining

The standard approach to vision-language pretraining, exemplified by the original CLIP model, uses a contrastive InfoNCE loss that normalizes similarity scores across the entire batch. While this approach has proven effective for learning joint embeddings, the softmax-based normalization introduces training instabilities and limits scalability as batch sizes grow. SigLIP, which stands for Sigmoid Loss for Language Image Pre-training, offers an alternative that has rapidly become the de facto standard for modern VLM vision encoders, addressing basic limitations in the contrastive learning paradigm.

Recall from Contrastive Learning for Retrieval that InfoNCE computes the probability of matching image-text pairs using a softmax normalization over all possible pairs within the batch:

$$L_{\text{InfoNCE}} = -\log \frac{\exp(s_{i,i}/\tau)}{\sum_{j} \exp(s_{i,j}/\tau)}$$

where:

• $s_{i,j}$: the similarity score between image $i$ and text $j$ (higher values indicate stronger match)
• $s_{i,i}$: the similarity between matching pair $i$ (the positive example)
• $\tau$: temperature parameter controlling the sharpness of the distribution (lower values create sharper distinctions)
• $\sum_{j} \exp(s_{i,j}/\tau)$: normalizing constant summing over all text candidates $j$ in the batch

This formulation computes the log-probability of selecting the correct text match from among all candidates in the batch. The temperature $\tau$ scales the similarities: when $\tau$ approaches 0, the distribution becomes sharp, approaching a one-hot selection of the maximum similarity; when $\tau$ approaches infinity, all probabilities become uniform.

However, the denominator requires computing similarities across the entire batch, creating a synchronization point in distributed training and potential instability when batch sizes grow large. The softmax normalization forces a competitive dynamic where improving the score of one positive pair necessarily affects the gradients of all other pairs in the batch. This global competition means that the gradient signal for any single pair depends on all other pairs in the batch, creating complex training dynamics that are difficult to control at scale.

SigLIP replaces this competitive normalization with a simple binary classification objective. Treating each image-text pair as an independent binary classification problem (match versus no-match), the loss becomes:

$$L_{\text{SigLIP}} = -\sum_{i,j} \left[ \mathbb{1}_{i=j} \log \sigma(s_{i,j}) + \mathbb{1}_{i \neq j} \log (1 - \sigma(s_{i,j})) \right]$$

where:

• $\mathbb{1}_{i=j}$: indicator function equal to 1 when image $i$ matches text $j$, 0 otherwise (identifies positive pairs)
• $\mathbb{1}_{i \neq j}$: indicator function equal to 1 when image $i$ does not match text $j$ (identifies negative pairs)
• $\sigma(\cdot)$: the sigmoid function $\frac{1}{1 + e^{-s}}$, mapping similarity scores to $(0, 1)$ probability range
• $s_{i,j}$: similarity score between image $i$ and text $j$
• $\sum_{i,j}$: sum over all image-text pairs in the batch

Unlike InfoNCE, this formulation treats each pair as an independent binary classification problem. The sigmoid outputs the probability that a given pair matches, with no normalization across other pairs required. This independence offers three key advantages for VLM encoders:

1. No batch normalization: Each pair's loss is independent, eliminating the costly softmax normalization across devices and letting larger effective batch sizes through gradient accumulation without synchronization barriers. In distributed training, this means each GPU can process its local batch without waiting for global statistics.

2. Stable gradients: The sigmoid's bounded output prevents the extreme gradient magnitudes that can occur with exponentials in large batches. When similarity scores become large in magnitude, the sigmoid saturates gracefully, whereas softmax can produce numerically unstable gradients.

3. Tunable bias: SigLIP introduces a learnable bias term $b$ in the scoring function $s_{i,j} = \langle v_i, t_j \rangle + b$, letting the model to learn an optimal decision threshold rather than assuming zero-centered similarities. This proves particularly valuable when the distribution of positive and negative pairs is imbalanced or when the model needs to adjust its confidence calibration.

One of the more subtle but important details in SigLIP's design is the treatment of its scaling parameters. In the CLIP formulation, the temperature $\tau$ is a learned scalar that controls the sharpness of the probability distribution. In SigLIP, two learnable scalars take its place: a log-temperature $\log t$ and a bias $b$. The full scoring formula becomes:

$$s_{i,j} = t \cdot \langle v_i, t_j \rangle + b$$

where:

• $t$: learned temperature, initialized to a value that produces reasonable initial logit magnitudes (often $\exp(10)$ for image-text settings)
• $\langle v_i, t_j \rangle$: cosine similarity between normalized image embedding $v_i$ and text embedding $t_j$
• $b$: learned bias, initialized to a small negative value such as $-10$ to prevent the model from predicting all pairs as matches at the start of training

The negative bias initialization is conceptually important. At the beginning of training, random embeddings produce near-zero cosine similarities. Without a bias, $\sigma(0) = 0.5$, meaning every pair starts with a 50% match probability. For a batch of 1024 images paired with 1024 texts, more than one million pairs exist, and only 1024 of them are true matches. Initializing the bias to $-10$ shifts the sigmoid output toward near-zero for all pairs. This provides a sensible prior that most pairs do not match. The model then learns to push true matches above the threshold while keeping false pairs below it, rather than starting from a confused state where everything looks like a match.

The temperature $t$ serves a similar calibration role. A high initial temperature means the model applies little scaling to similarities, creating soft predictions that allow gradient flow even for near-zero similarities early in training. As training progresses and the encoder produces more discriminative embeddings, the effective temperature can be tuned to sharpen predictions where the model is confident and maintain softness where ambiguity exists.

Despite the different training objective, SigLIP uses the same Vision Transformer architecture as CLIP for the vision encoder. The improvements come primarily from the training dynamics and the shape of the loss surface rather than structural changes. However, SigLIP-trained encoders tend to produce embeddings with better calibration for downstream VLM use, where the vision encoder often remains frozen while the language model adapts to process visual inputs.

The sigmoid loss particularly benefits scenarios with noisy training data, which is common in large-scale web datasets like LAION or COYO-700M. Because SigLIP does not force hard contrasts between all pairs in a batch, it can handle ambiguous pairs that might be weak matches, such as an image that contains multiple concepts or text descriptions that partially match the visual content. This flexibility allows the encoder to preserve fine-grained semantic relationships that competitive normalization might collapse. When an image depicts both a dog and a park bench, an InfoNCE model must decide which of these descriptions is the correct match and push all others toward zero probability. A SigLIP model can assign moderate probability to both the dog description and the park bench description without loss divergence, capturing the ambiguity in the scene.

In practice, the most widely deployed SigLIP configuration is the SigLIP-SO400M family, where SO stands for "Scaled-up by Oversized" and 400M refers to the approximately 400 million parameter vision encoder. This family has become the default vision backbone in several prominent VLMs including PaLI-Gemma, LLaVA successors, and various open-source multimodal systems.

The SigLIP-SO400M-14/384 configuration uses 14x14 patches at 384x384 resolution, creating 729 patch tokens per image. This configuration represents a deliberate compromise: patch size 14 offers finer granularity than the original ViT-B/16 configuration, and the 384x384 resolution captures enough detail for most practical tasks without the memory costs of 512x512 or higher. The "SO" scaling procedure increases the model's depth and width beyond the standard ViT-L specification by applying compound scaling, distributing additional capacity across attention heads, hidden dimensions, and the number of transformer layers simultaneously.

The embeddings produced by SigLIP-SO400M at 384x384 resolution support a remarkably broad range of visual tasks out of the box. Natural image understanding, chart and diagram interpretation, and basic document reading all work well with this configuration as a frozen encoder. This breadth reflects the advantages of training on diverse, web-scale image-text pairs with a stable loss function that preserves multi-concept relationships rather than forcing each image into a single semantic category.