Vision Transformer › Key Parameters and Model Variants

The key parameters for Vision Transformer configurations, along with their practical implications, are:

• img_size: Size of the input image in pixels (assumes square). Controls how many patches are produced; larger images create more patches and increase attention computation quadratically.
• patch_size: Side length of each square patch in pixels. The standard choice is 16. This provides 196 patches for a $224 \times 224$ image. ViT-H uses 14 to give the largest model finer spatial resolution.
• embed_dim: Embedding dimension $D$. Common values are 192 (Tiny), 384 (Small), 768 (Base), 1024 (Large), and 1280 (Huge). This is the primary driver of model width and parameter count.
• depth: Number of transformer blocks $L$. Ranges from 4 in lightweight configurations to 32 in ViT-Huge. Deeper models have more capacity for compositional reasoning but require more memory and computation.
• num_heads: Number of attention heads. Must evenly divide embed_dim. More heads allow the model to attend to different representation subspaces simultaneously, with each head specializing in different spatial relationships or feature types.
• mlp_ratio: Ratio of the MLP hidden dimension to embed_dim. Standard is 4.0, giving a hidden dimension of $4D$ in the feed-forward layers. Higher ratios increase model capacity and parameter count.
• drop_rate: Dropout probability applied to position embeddings, MLP activations, and attention weights. Provides regularization during fine-tuning on smaller datasets.