Active Perception, Dual Control, and Exploration › Curiosity and Intrinsic Motivation

Information gain is a property of an action given a belief. Curiosity is an agent-level disposition: a tendency to choose such actions, especially when extrinsic reward is sparse or absent. In reinforcement learning, we implement curiosity by adding an intrinsic reward $r^{\text{int}}_t$ to the environment's extrinsic reward $r^{\text{ext}}_t$ and optimizing the sum (or a discounted combination). The intrinsic term is a hand-designed or learned signal that rewards the agent for experiencing informative or novel states.

A useful mental model treats the intrinsic reward as a subsidy. The environment pays the agent for task progress, but the task is often so sparse that the agent never finds the first payment. The intrinsic reward is a short-term subsidy that keeps the agent searching. The subsidy is not free: it changes the incentives the agent sees, and if it is paid forever the agent may spend its whole life collecting it. The engineering problem of intrinsic motivation is to design a subsidy that helps the agent find the real reward and then gets out of the way.

Two complementary motivations drive intrinsic rewards.

1. Statistical: In sparse-reward environments, few sampled trajectories contain nonzero returns, which makes credit assignment and gradient estimation difficult. An intrinsic reward supplies a denser learning signal.
2. Computational hypothesis: organisms do explore without immediate task reward, and computational accounts model curiosity as learning or compression progress. This is a hypothesis about a useful intrinsic objective, not an established evolutionary explanation.

World models make the statistical case concrete: coverage of decision-relevant state-action regions is one requirement for reliable planning, alongside identifiability, a suitable model class, and robust decision making under residual error. Intrinsic reward can improve the data used by the model, not just help the policy stumble onto task reward. This connects to [Online, Offline, and Continual Learning].

Intrinsic rewards differ in what they reward. The main families are:

• Count-based: reward $1 / \sqrt{N(s,a)}$ or a pseudo-count analogue. Density surprisal $-\log \hat p(s)$ is related but distinct. Optimistic model-based algorithms such as R-MAX also use visitation thresholds, but they do not implement this inverse-count reward directly.
• Prediction-error (surprise): reward $-\log \hat{P}(s'_t \mid s_t, a_t)$. This is a surprise bonus, not information gain in general: irreducible stochasticity can keep it high after there is nothing left to learn, producing the noisy-TV problem.
• Ensemble disagreement: reward spread across diverse model predictions. With calibrated members and aleatoric uncertainty modeled separately, disagreement can serve as an epistemic proxy; those conditions are not automatic.
• Information-theoretic / Bayesian surprise: reward the realized posterior-to-prior KL divergence after an observation. Its expectation over observations is mutual information; the realized quantity need not equal the expected entropy reduction for a particular outcome.
• Empowerment: maximize channel capacity from action sequences to future states, commonly written $\max_{p(a_{t:t+k-1}\mid s_t)} I(A_{t:t+k-1};S_{t+k}\mid s_t)$. This asks how many distinguishable future outcomes the agent can control and connects to the skills in [Hierarchical, Symbolic, Language, and Multi-Agent Planning].
• Novelty in learned latent space: reward the novelty of a state's embedding under an encoder, which avoids counting in raw dimensions.

A useful way to organize this taxonomy is by what is being predicted: the agent can be curious about states (visit novel states), about transitions (learn the dynamics), about parameters (identify the system, cf. [System Identification]), or about its own abilities (discover skills). These reward different things and can be combined.

Suppose you want an agent to learn a manufacturing line that occasionally jams. A count bonus tends to spread visits, a prediction-error bonus may overvalue the irreducibly random jam, and a calibrated ensemble may focus on regions where plausible dynamics models disagree. An information-theoretic objective instead rewards observations that update a named belief about the line. These are intended tendencies, not deterministic outcomes: representation, calibration, optimization, and bonus scale can reverse them.

The standard formalization of the tradeoff is the regret of a bandit or MDP. For a multi-armed bandit with $K$ arms and mean rewards $\mu_i$, let $\mu^* = \max_i \mu_i$ be the best-arm mean. The regret after $T$ pulls is

$$\mathrm{Regret}(T) = \sum_{t=1}^{T} \left( \mu^* - \mu_{a_t} \right),$$

where:

• $K$: the number of arms
• $\mu_i$: the unknown mean reward of arm $i$
• $\mu^* = \max_i \mu_i$: the mean reward of the best arm
• $a_t \in \{1, \dots, K\}$: the arm pulled at time step $t$
• $\mu_{a_t}$: the mean reward of the arm chosen at step $t$
• $T$: the total number of pulls, or horizon

This quantity is pseudo-regret relative to always pulling the arm with the highest mean reward, not realized hindsight regret. Sublinear pseudo-regret means average per-step regret goes to zero as $T$ grows. A policy that permanently commits to a suboptimal arm with gap $\Delta>0$ incurs $\Delta T$ pseudo-regret.

• $\varepsilon$-greedy: act greedily with probability $1-\varepsilon$, randomly otherwise. With fixed $\varepsilon$, exploration neither vanishes nor responds to arm-specific uncertainty. A decaying schedule can use time, but remains externally specified unless it is tied to uncertainty.
• Optimism (UCB1): after pulling every arm at least once, choose $\arg\max_i [\hat{\mu}_i(t)+\sqrt{2\log t/N_i(t)}]$. For independent stationary arm rewards normalized to $[0,1]$, UCB1 has logarithmic instance-dependent expected regret. For rewards in a known interval $[a,b]$, normalize them or scale the confidence radius by $b-a$.
• Thompson sampling: sample a reward estimate from the posterior and act greedily on the sample. This gracefully balances exploration and exploitation because widely uncertain arms get sampled high some of the time.
• Information-directed sampling (IDS): choose an action distribution that minimizes squared expected one-step regret divided by mutual information about the optimal action. Information-ratio analyses yield Bayesian regret bounds for broad model classes; the statement is not an unconditional near-optimality guarantee.

For MDPs, related ideas use state-dependent counting, ensembles, uncertainty propagation, conservative penalties, or short synthetic rollouts. These mechanisms are not interchangeable: PETS propagates probabilistic ensemble predictions while optimizing task reward, whereas MBPO limits model bias with short model rollouts. Part VIII develops those distinctions.

For standard policy-gradient, actor-critic, or value-based learning, an intrinsic reward only needs to be computable as a scalar; the optimizer does not differentiate through the reward generator or environment. Differentiability is additionally useful when gradients are propagated through a learned model or planner. Two standard constructions are:

RND-style (random network distillation). Fix a randomly initialized network $f_\theta$ and train a predictor $g_\phi$ to match its output on visited states. The intrinsic reward is $\|g_\phi(s) - f_\theta(s)\|^2$. Novel states often have higher prediction error when the predictor does not generalize to them. The mismatch between the fixed random target and the learned predictor is a stand-in for surprise.

Latent-space disagreement. Train a diverse ensemble $\{h_k\}$ of encoders or dynamics models. The intrinsic reward is the variance or Jensen-Shannon divergence across comparable member predictions. Its epistemic interpretation depends on diversity, calibration, and separate modeling of observation noise.

Both compose with standard policy optimizers. A practical failure is a persistent prediction-error bonus: the agent finds observations that remain hard for the predictor for reasons unrelated to useful novelty and farms them. The residual may be driven by background clutter, sensor noise, limited predictor capacity, or nonstationarity rather than by reducible uncertainty.

Consider a toy robot in a room with a light switch. Pressing it may initially be informative about the lighting model. As evidence identifies the causal link, Bayesian information gain about that parameter falls. An exact count bonus also decays with visits, but its decay tracks visitation rather than whether the causal parameter has been learned. A prediction-error agent can still be fooled by random flicker because irreducible surprise remains high.

The example shows why novelty differs from information. Count novelty decreases as a state is revisited; information gain decreases as uncertainty about a specified latent variable is resolved. The two can fall at different rates and can generalize differently across representations. Decision value adds a third test: even information about the named variable may not change the task action.

The intrinsic weight $\beta$ is a first-class hyperparameter. Too small and curiosity may not change behavior; too large and the policy may optimize the proxy instead of the task. One option is to anneal $\beta$; others normalize the two returns or learn separate value heads. A multiplicative form such as $r^{\text{ext}}(1+r^{\text{int}})$ is task-dependent: it erases exploration wherever $r^{\text{ext}}=0$ and behaves poorly when rewards change sign, so it provides no general protection against intrinsic-reward domination.

We now have an information objective and a motivation to pursue it. The remaining piece is making sure the pursuit is safe.