Online, Offline, and Continual Learning › Key Parameters

The key parameters for online and continual learning are:

• Learning rate ($\eta$): Controls the memory horizon of streaming updates. Larger values adapt faster but enlarge the stationary noise ball, which shrinks only as $\sqrt{\eta}$.
• Forgetting factor ($\lambda$): Discounts older observations in recursive least squares. Smaller values shorten the effective window and inflate covariance to preserve plasticity.
• Buffer size: Sets how many past transitions replay retains. Larger buffers preserve a broad distribution but slow adaptation to real change.
• Replay fraction: The proportion of each batch drawn from the buffer rather than fresh data. Higher values stabilize learning but reduce responsiveness.
• Update-to-data ratio: The number of gradient steps per environment step collected. High ratios extract more from each interaction but risk overfitting to stale data.
• EWC regularization strength ($\lambda_{\mathrm{EWC}}$): Controls how strongly the Fisher-weighted penalty anchors important parameters against change during new tasks.

These parameters are not independent. The learning rate and the replay fraction both control how quickly the model responds to new data. The buffer size and the forgetting factor both control how much history is retained. The update-to-data ratio and the EWC strength both control how strongly the model is pushed toward the current task. Tuning one without considering the others can produce surprising results, because the parameters interact. A high replay fraction with a high learning rate may be more adaptive than a low replay fraction with a low learning rate, even though both changes push in the same direction.