Architecture Methods › HAT: Hard Attention to the Task

Hard Attention to the Task (HAT, Serra et al. 2018) is a mask-based architecture method that falls conceptually between expansion and progressive approaches. Rather than growing the network, HAT learns a binary mask over network units for each task and uses these masks to protect important units from modification during future tasks.

The key innovation in HAT is the way it learns these masks end-to-end during task training, rather than deriving them post-hoc (as in PackNet, which prunes after training). This means the mask for each task is shaped by the actual gradient signal on that task's objective, making sure that the protected units are the ones most useful for the task rather than simply the ones with the largest weights.

HAT assigns each task $t$ a learnable embedding vector $\mathbf{e}_t \in \mathbb{R}^L$ where $L$ is the number of layers. For each layer $l$, an attention vector $\mathbf{a}_l^t \in [0, 1]^{d_l}$ is produced by passing the task embedding through a sigmoid:

$$\mathbf{a}_l^t = \sigma\!\left( s \cdot \mathbf{e}_t^l \right)$$

where:

• $\mathbf{e}_t^l$: the $l$-th component of the task embedding for task $t$
• $s$: a temperature parameter controlling the sharpness of the mask (large $s$ pushes values toward 0 or 1)
• $\sigma$: the sigmoid function

During the forward pass for task $t$, the activation at each layer $l$ is multiplied element-wise by $\mathbf{a}_l^t$, effectively gating which units contribute to the computation for that task. At training time, $s$ starts small (producing soft, continuous masks near 0.5) and is annealed to large values so that at the end of training, each mask entry is approximately binary. At the start, the soft masks allow gradients to flow through all units, making the optimization landscape smooth. At the end, the hard masks provide crisp unit-level protection boundaries.

This annealing strategy prevents gradients from vanishing before the model has learned which units to claim. If you trained with hard binary masks from the start, the gradient would vanish for units where the mask is exactly zero. By starting soft and hardening gradually, HAT maintains trainability throughout while converging to a clean partition.

The mechanism that protects prior tasks is gradient masking. Before updating parameters during training on task $t$, HAT computes a cumulative mask $\mathbf{m}_l^{\text{cum}}$ that is the element-wise maximum of all previous tasks' hard masks:

$$\mathbf{m}_l^{\text{cum}} = \max_{t' < t} \mathbf{a}_l^{t'}$$

This cumulative mask identifies units that are important to at least one prior task. Gradient updates to these units are then scaled down by a factor proportional to $1 - \mathbf{m}_l^{\text{cum}}$, effectively freezing units that have been claimed by prior tasks while allowing unclaimed units to be freely modified.

The result is that each task carves out its own subset of network capacity, and the union of all prior task masks acts as an expanding write-protection region. New tasks must work within the remaining unprotected capacity.

This gradient manipulation happens outside the standard backpropagation graph. After computing gradients through standard backprop, HAT applies an element-wise scale to the gradient tensor before the optimizer step. This is a direct surgical intervention: you are not changing the loss function but instead controlling which gradient directions the optimizer is allowed to follow. The effect is mathematically equivalent to projecting the gradient onto the subspace orthogonal to the protected units.

HAT has a key advantage over progressive networks: the network size is fixed. The overhead per task is a single mask vector per layer, which is $O(d_l)$ per layer per task rather than $O(N)$ per task. This is much more memory-efficient. The limitation is that capacity for new tasks shrinks as more masks accumulate. HAT includes a capacity penalty in the training objective to encourage sparse masks that leave room for future tasks:

$$\mathcal{L}_{\text{HAT}}^{(t)} = \mathcal{L}_{\text{task}}^{(t)} + \lambda \sum_{l} \left\| \mathbf{a}_l^t \odot (1 - \mathbf{m}_l^{\text{cum}}) \right\|_1$$

where:

• $\lambda$: sparsity penalty coefficient
• $\odot$: element-wise multiplication
• The $L_1$ norm encourages the new task's attention to be sparse within the unclaimed region, preserving capacity for tasks that have not yet arrived

The interaction between the two terms in this loss is subtle. The task loss wants to activate many units to achieve good performance. The sparsity penalty wants to activate as few unclaimed units as possible. These are in tension, and the parameter $\lambda$ controls the balance. A large $\lambda$ forces extreme sparsity, potentially degrading per-task accuracy but preserving more capacity for future tasks. A small $\lambda$ allows generous unit allocation per task, improving current accuracy but potentially exhausting capacity sooner.

HAT was evaluated on a sequence of up to eight different classification datasets in the original paper, including permuted MNIST variants, CIFAR-10, and others. The results demonstrated near-zero backward transfer (forgetting) while maintaining competitive forward accuracy. The masks tended to use roughly 20-40% of units per task depending on task complexity, which means a network can realistically support around five to eight tasks before the unclaimed region becomes too small to reliably accommodate a new one.

One practical extension of HAT is to grow the network when the unclaimed fraction falls below a threshold. This hybrid approach combines the mask-based protection of HAT with the growth capability of expansion methods: you use HAT as long as there is sufficient unclaimed capacity, and you expand the backbone when the unclaimed region runs low. This is more practical than pure expansion (which would grow indefinitely) and more scalable than pure HAT (which would degrade on task ten-plus).