DeBERTa › Key Parameters

When working with DeBERTa, these parameters most significantly affect performance and efficiency:

• max_relative_positions (default: 512): The maximum relative distance encoded with unique embeddings. Positions beyond this distance share boundary embeddings. Larger values capture finer-grained positional information but increase memory for position embeddings quadratically with sequence length.

• hidden_size (768 for Base, 1024 for Large): The dimension of hidden representations. DeBERTa follows BERT's hidden size conventions. Larger hidden sizes increase model capacity but also multiply the cost of disentangled attention proportionally.

• num_heads (12 for Base, 16 for Large): Number of attention heads. Each head computes three attention components (c2c, c2p, p2c), so more heads increase both expressiveness and computation. The disentangled attention formulation means that each head operates on separate content and position subspaces of dimension hidden_size / num_heads.

• pos_att_type (default: ["c2p", "p2c"]): Which disentangled attention components to include. Including both components (the default) yields the best results, but ablation studies have shown that either component alone outperforms no positional attention.

• emd_layers (default: 2): Number of Enhanced Mask Decoder layers. These additional transformer layers incorporate absolute position information before MLM prediction. Two layers is the sweet spot found by the DeBERTa authors.

• relative_attention (default: True): Whether to use relative position attention throughout the encoder. When set to False, the model falls back to absolute position encoding like BERT, losing the disentangled attention benefits but reducing computational cost.