Full Fine-tuning › Worked Example: Configuring Fine-tuning from Scratch

To build concrete intuition for how these hyperparameters interact, let's walk through a complete configuration calculation from scratch. Suppose we want to fine-tune BERT-base on a sentiment classification dataset with 8,000 training examples, using a batch size of 16 and planning to train for 3 epochs.

Step 1: Calculate total training steps. Each epoch processes all 8,000 training examples in batches of 16.

$$\begin{aligned} \text{steps\_per\_epoch} &= \lceil \text{train\_size} / \text{batch\_size} \rceil \\ &= \lceil 8000 / 16 \rceil \\ &= 500 \end{aligned}$$

With 3 epochs, the total number of training steps is:

$$\begin{aligned} T_{\text{total}} &= \text{steps\_per\_epoch} \times \text{num\_epochs} \\ &= 500 \times 3 \\ &= 1500 \end{aligned}$$

Step 2: Calculate warmup steps. With a warmup ratio of 0.1, we devote 10% of training to warmup:

$$\begin{aligned} T_{\text{warmup}} &= \lfloor T_{\text{total}} \times \text{warmup\_ratio} \rfloor \\ &= \lfloor 1500 \times 0.1 \rfloor \\ &= 150 \end{aligned}$$

So the learning rate increases linearly over the first 150 steps, then decays over the remaining 1,350 steps.

Step 3: Verify the warmup-to-training ratio. The warmup covers 150 out of 500 steps in the first epoch, meaning the learning rate is still ramping up for the first 30% of epoch 1. This is normal and expected. By step 150 (30% into epoch 1), the learning rate reaches its maximum. It then decays linearly over the remaining 1,350 steps.

Step 4: Calculate the learning rate at each step. Using $\text{lr}_{\text{max}} = 2 \times 10^{-5}$ and the linear decay formula, the learning rate at step 750 (halfway through training, midpoint of the decay phase) is:

$$\begin{aligned} \text{lr}(750) &= \text{lr}_{\text{max}} \times \left(1 - \frac{750 - 150}{1500 - 150}\right) \\ &= 2 \times 10^{-5} \times \left(1 - \frac{600}{1350}\right) \\ &= 2 \times 10^{-5} \times (1 - 0.444) \\ &= 2 \times 10^{-5} \times 0.556 \\ &= 1.11 \times 10^{-5} \end{aligned}$$

At the midpoint of the decay phase, the learning rate has decreased to approximately $1.11 \times 10^{-5}$, which is about 55.6% of its maximum value. This is expected from a linear schedule: at the midpoint, you should have approximately half the maximum learning rate remaining.

Step 5: Calculate effective batch size with gradient accumulation. If GPU memory allows only 8 samples per forward pass but we want an effective batch size of 32, we need gradient accumulation:

$$\begin{aligned} N_{\text{accum}} &= B_{\text{effective}} / B_{\text{device}} \\ &= 32 / 8 \\ &= 4 \end{aligned}$$

With gradient accumulation of 4, the number of optimizer steps per epoch becomes:

$$\begin{aligned} \text{optimizer\_steps\_per\_epoch} &= \text{steps\_per\_epoch} / N_{\text{accum}} \\ &= 500 / 4 \\ &= 125 \end{aligned}$$

This means the total number of parameter updates across all 3 epochs is $125 \times 3 = 375$, rather than the 1,500 forward/backward passes. The gradient quality is equivalent to using a batch size of 32, but you need only 8 samples in GPU memory at any time.

Step 6: Verify the linear scaling rule for learning rate. Our baseline learning rate of $2 \times 10^{-5}$ was validated for batch size 16. If we switch to effective batch size 32, the linear scaling rule suggests:

$$\begin{aligned} \text{lr}_{\text{new}} &= \text{lr}_{\text{base}} \times \frac{B_{\text{new}}}{B_{\text{base}}} \\ &= 2 \times 10^{-5} \times \frac{32}{16} \\ &= 4 \times 10^{-5} \end{aligned}$$

However, since we are using gradient accumulation to simulate batch size 32 rather than true batch size 32, the effective gradient dynamics are slightly different. In practice, gradient accumulation with 4 steps of batch size 8 is mathematically equivalent to true batch size 32, so the learning rate scaling applies equally. The final configuration for this scenario would be: batch size 8, gradient accumulation 4, effective batch size 32, learning rate $4 \times 10^{-5}$, warmup steps 150, total optimizer steps 375.