ALBERT › Different Sharing Strategies
Language AI HandbookAll chapters ALBERT › Different Sharing StrategiesOne section · Sections in this chapter6 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
ALBERT's paper explored several sharing strategies beyond full sharing. You can share only attention parameters, only feed-forward parameters, or share everything. The experiments revealed that full sharing works surprisingly well.
def compare_sharing_strategies(
hidden_dim, num_heads, intermediate_dim, num_layers
):
"""Compare parameter counts under different sharing strategies."""
# Per-layer component sizes
attention_params = (
4 * hidden_dim * hidden_dim + 4 * hidden_dim
) # Q,K,V,O projections
attention_norm = 2 * hidden_dim
ffn_params = (
hidden_dim * intermediate_dim + intermediate_dim
) # Up projection
ffn_params += intermediate_dim * hidden_dim + hidden_dim # Down projection
ffn_norm = 2 * hidden_dim
attention_total = attention_params + attention_norm
ffn_total = ffn_params + ffn_norm
layer_total = attention_total + ffn_total
strategies = {
"No sharing (BERT)": {
"attention": attention_total * num_layers,
"ffn": ffn_total * num_layers,
},
"Share attention only": {
"attention": attention_total, # Shared
"ffn": ffn_total * num_layers, # Unique
},
"Share FFN only": {
"attention": attention_total * num_layers, # Unique
"ffn": ffn_total, # Shared
},
"Share all (ALBERT)": {
"attention": attention_total, # Shared
"ffn": ffn_total, # Shared
},
}
for name, params in strategies.items():
params["total"] = params["attention"] + params["ffn"]
return strategies, layer_total * num_layersLayer Parameter Comparison (hidden=768, 12 layers): ------------------------------------------------------- No sharing (BERT) | 85,054,464 | 0.0% reduction Share attention only | 59,051,520 | 30.6% reduction Share FFN only | 33,090,816 | 61.1% reduction Share all (ALBERT) | 7,087,872 | 91.7% reduction
Full sharing provides the largest reduction. The ALBERT paper showed that performance degradation from sharing was minimal, particularly for the feed-forward layers. Attention sharing has slightly more impact because different layers benefit from attending to different aspects of the input.
