Architecture Methods › Key Parameters

Architecture-based continual learning involves a richer set of hyperparameters than regularization or replay methods, because you are making decisions about the training objective and the network's physical structure. Understanding which parameters control which trade-offs is essential for applying these methods effectively.

The key parameters for architecture-based continual learning methods are:

• hidden_dim: The total width of the shared network backbone. Larger hidden layers provide more capacity for mask-based methods like HAT to allocate across tasks. For progressive networks, this controls the per-column capacity. Increasing width helps both with per-task accuracy and with the number of tasks that can fit, but at the cost of a larger total model size.
• lateral connections: In progressive networks, the lateral weight matrices are initialized to zero so each column begins independent. Non-zero initialization would allow immediate feature borrowing but risks early-task representations dominating the new column before it has established its own features. The lateral connection rank (if using low-rank adapters) controls the expressivity of cross-column communication.
• s_max: The maximum temperature for HAT mask annealing. Higher values produce harder (more binary) masks but can make gradient descent more difficult near the end of training as the soft mask values saturate. Values in the range 50 to 400 have been reported in practice, with the optimal choice depending on network size and task complexity.
• lam (lambda): The sparsity penalty coefficient in HAT. Higher values force sparser masks, preserving more capacity for future tasks at the expense of potentially using fewer neurons for the current task and reducing per-task accuracy. This parameter effectively controls the future-versus-present trade-off: a high lambda is more conservative about current resource consumption.
• pruning_rate (for PackNet-style methods): The fraction of weights freed after each task. Higher rates free more capacity for future tasks but give each task fewer parameters to work with, harming accuracy on tasks trained later in the sequence. The optimal pruning rate is a function of the total number of tasks anticipated, which must often be estimated in advance.
• expansion threshold (for DEN-style methods): The loss threshold above which the network adds new neurons. A lower threshold triggers expansion more aggressively, giving better per-task accuracy but growing the network more quickly. Setting this threshold requires understanding the typical loss values on convergence for tasks of the expected difficulty.
• adapter rank (for LoRA and adapter methods): Controls the expressiveness of the task-specific delta. Larger rank allows more complex task adaptations but increases per-task storage. Typical values range from 4 to 128 depending on model size and task complexity.