BERT Architecture › Practical Considerations

When deploying BERT, two factors dominate resource requirements: memory consumption and computational cost. Both scale with sequence length, making long documents particularly expensive to process. Understanding these scaling properties is essential for making informed decisions about batch sizes, sequence length truncation, and hardware requirements before you begin a project.

BERT's memory usage during inference scales with sequence length squared due to the attention mechanism. For a batch of sequences, the dominant terms are:

• Embeddings: $O(\text{batch} \times \text{seq} \times d)$
• Attention scores: $O(\text{batch} \times \text{heads} \times \text{seq}^2)$
• Intermediate activations: $O(\text{batch} \times \text{seq} \times 4d)$

For BERT-Base with a sequence length of 512 and batch size of 1, the attention scores alone require approximately 12 × 512 × 512 × 4 bytes (float32) = 12.6 MB per layer, or about 150 MB for all 12 layers.

The computational complexity of BERT is dominated by three operations:

1. Attention: $O(n^2 \cdot d)$ for sequence length $n$ and dimension $d$
2. Feed-forward: $O(n \cdot d^2)$ with the 4x expansion
3. Embeddings: $O(n \cdot d)$

For typical sequence lengths (128-512), attention and feed-forward costs are comparable. The quadratic attention cost becomes prohibitive only for very long sequences, motivating efficient attention variants like Longformer and BigBird.

The crossover point between attention and feed-forward dominance is worth calculating explicitly. Attention cost scales as $O(n^2 d)$ and feed-forward scales as $O(n d^2)$, so attention dominates when $n > d$. For BERT-Base with $d = 768$, attention dominates for sequences longer than 768 tokens, which is already beyond BERT's 512-token limit. This means that for all practical BERT inputs, attention and feed-forward computation are similar in magnitude, and reducing sequence length reduces both approximately linearly.

In practice, most fine-tuning workloads for classification tasks truncate sequences to 128 tokens rather than using the full 512. At 128 tokens, BERT runs roughly 16 times faster than at 512 tokens (due to the quadratic attention term, a 4x reduction in length gives a 16x reduction in attention compute), with only modest performance degradation for tasks whose inputs fit in 128 tokens. This truncation is the single most impactful optimization you can make when deploying BERT in a latency-constrained environment.