Active Perception, Dual Control, and Exploration

A robot at a warehouse intersection has two equally reachable pallets in front of it. It has never visited this aisle before. Every sensor reading so far has been coarse and noisy. The pallet labels are unreadable until the robot rotates its camera and closes the distance, and the aisle layout is only half-known. If the robot simply drives to whichever pallet looks closer and picks it up, it sometimes grabs the wrong item and pays a penalty. If it instead spends a few seconds rotating and peeking down the aisle, it uses battery and time but learns which pallet holds the target. Neither pure exploitation (act on what you believe right now) nor pure exploration (wander until the world is known) is obviously correct. The right move depends on how much a little information would change the agent's optimal action.

That tension is the subject of this chapter. So far in Part VII: Planning and Agency, we have treated planning as a problem of choosing actions that maximize predicted future reward. Search and Belief-Space Planning already made uncertainty part of the planning state, and Policies Learned in Imagination trained a policy inside a learned model. Here we focus on the next question: when should the agent choose an action partly because of what that action will reveal?

The central idea is dual control, introduced by Feldbaum in 1960: a control action can serve two purposes at once. It can steer the system toward reward and it can probe the system to reduce uncertainty about its state or parameters. An action can therefore have both control value and information value; whether it is preferable depends on the belief, horizon, cost of delay, and later decisions that can use the information. Active perception is a closely related setting in which the agent controls how it senses---by moving a camera, refocusing, touching, or querying---sometimes jointly with task actions.

A treatment of world models devotes a chapter to this topic because a learned world model is only as trustworthy as the data used to train it, and the data an agent collects is shaped by the actions it chooses. An agent that never probes can remain accurate only along its visited state-action tube and may plan confidently outside its data support unless pretraining, off-policy data, model structure, or calibrated uncertainty provides broader coverage. The feedback loop between acting, learning, and planning is the reason exploration cannot always be treated as a preprocessing step. Deciding what to do next and deciding what to learn next are often coupled decisions.

By the end of this chapter you will be able to

• Reason about actions as carriers of both reward and information, and recognize when these two roles reinforce each other and when they conflict.
• Define and compute information gain in the standard Shannon sense, and connect it to uncertainty reduction through the belief-MDP and the mutual-information identity.
• Distinguish information-seeking (intrinsic) objectives from extrinsic reward, understand why the combination is delicate, and identify the characteristic failure modes of each intrinsic-reward family.
• Explore in the presence of safety constraints, where not every informative action is admissible, and state the assumptions that make safety guarantees valid.

We begin with the concept that makes all of this precise: ascribing value to information.

Before we can say an action "improves knowledge," we need a quantity that measures knowledge and a way to say an action changes it. The prior chapter on Bayesian Filtering and Belief States already gave us the first: the agent maintains a belief $b(s)$ over the state, and actions plus observations update that belief. The quantity we will use to compare beliefs is the entropy of the belief. But to make the reward and knowledge objectives comparable, we need to frame the whole problem as a single decision problem, and for that the belief must be part of the state the planner reasons over.

This section builds that framing in three steps. First we review the belief-MDP, the object that turns "knowledge" into something a planner can value like any other state feature. Then we introduce Feldbaum's dual effect, the phenomenon that makes probing a control problem rather than a bookkeeping nuisance, and we are careful about exactly when the effect is present. Finally we write down two ways to tuck information into the objective, one tractable and approximate, one exact but intractable, and we are explicit about the gap between them.

Recall from [Markov and Partially Observable Decision Processes] that a partially observable Markov decision process (POMDP) has states, actions, observations, a transition model $T(s' \mid s, a)$, an observation model $O(o \mid s', a)$, and a reward $R(s, a)$. When the true state is not known, the agent acts on its belief $b$, the posterior distribution over the state given the history. The key structural fact is that, under the assumption that the underlying state is Markov, the belief itself is a sufficient statistic for the history and evolves in a Markov way. Formally, the belief $b$ plays the role of a state in a new, fully observable process called the belief-MDP (also written belief-state MDP or B-MDP). Its transition is deterministic given an observation. Bayes' rule for the belief update is:

The denominator $P(o \mid b, a)$ is the predictive observation probability;

This denominator is what makes the belief-MDP legitimate: it is the normalizer that keeps the posterior a valid distribution, and as we will see, it is also the engine of information gain.

where:

• $b(s)$: the current belief probability assigned to state $s$ before action $a$ is taken
• $T(s' \mid s, a)$: the transition probability of moving from state $s$ to state $s'$ under action $a$
• $O(o \mid s', a)$: the observation probability of seeing $o$ after landing in $s'$ under action $a$
• $b'(s')$: the posterior belief probability assigned to state $s'$ after seeing observation $o$
• $P(o \mid b, a)$: the predictive probability of observation $o$ under belief $b$ and action $a$, which normalizes the posterior

The belief $b(s)$ is a probability, so it is nonnegative and sums to one; it encodes everything the agent knows about the state given its history. The transition $T$ describes how the world moves in the absence of observations, and the observation model $O$ describes how the world reveals itself. The two combine, through Bayes' rule, into the posterior $b'(s')$, which is what the agent believes after seeing $o$. Because $b'$ is a deterministic function of $(b, a, o)$ and $o$ is random, the belief-MDP is a fully observable stochastic process whose "state" is the belief. In other words, the randomness in the belief-MDP comes entirely from the observation $o$; once you fix the belief, the action, and the observation, the next belief is fixed. This property is what allows us to speak of the belief as a state in the ordinary MDP sense.

\newpage

The reason to build this object is that it lets us talk about "value of information" through ordinary continuation value: an action can move the belief to a region where later choices earn more reward. The belief-MDP is an exact reformulation when the model is known and the belief is maintained exactly. In general, the unrestricted space of distributions over a continuous state is infinite-dimensional. Special closed families, such as Gaussian beliefs in a linear-Gaussian model, have finite-dimensional sufficient parameters; outside such cases, practical methods approximate the belief or its value.

Feldbaum's dual control is the observation that in a stochastic control problem the action $a_t$ affects both the future distribution of states and the future distribution of beliefs about those states. These are the two "effects." The first is the ordinary control effect: $a_t$ changes the next-state distribution $p(s_{t+1} \mid b_t, a_t) = \sum_s b_t(s) T(s_{t+1} \mid s, a_t)$. The second is the probing and learning effect: different actions produce different observation likelihoods $P(o \mid b, a)$, so they lead to different posteriors $b_{t+1} = \tau(b_t, a_t, o_t)$ and different future uncertainty.

The probing effect is not always present. In standard linear-quadratic-Gaussian (LQG) control, the dynamics and observations are known and linear, process and observation noise are additive Gaussian and independent of control, and the objective is an expected quadratic cost without control-dependent sensing or hard constraints. Under those assumptions, the separation principle holds: the optimal controller feeds the Kalman conditional mean into the full-state linear-quadratic controller. The mean is also the MAP estimate in the Gaussian case. The covariance follows a Riccati recursion determined by the model and noise covariances, not by the realized control inputs, and the Kalman gains are computed from that covariance. The controller therefore has no incentive to probe merely to improve estimation.

The separation property is special, not generic. A dual effect can arise when actions alter the information available to later decisions, for example when:

• Nonlinear dynamics or state-dependent sensors make posterior uncertainty depend on where actions move the system.
• Control enters the state or observation channel multiplicatively, making the signal quality action-dependent.
• Unknown parameters are identifiable only under particular inputs, so controls determine how quickly the system can be learned.

None of these features is sufficient by itself. A fully observed nonlinear system has no hidden state to probe, and an unknown but unidentifiable parameter cannot be learned by any action. Reward and safety constraints determine whether an informative probe is worth taking or admissible; they do not create the information channel.

For the rest of this chapter we focus on regimes in which actions change decision-relevant uncertainty, as many online, partially observed world-model agents do.

The practical consequence is conditional rather than universal. Many online world-model agents do operate in a dual-effect regime because their actions determine future observations and parameter updates. A fixed learned model used in a fully observed task does not: learning provenance alone does not create a probing effect. The defining test is whether a current action changes uncertainty that matters to later decisions.

There are two broad ways to write down the objective once we accept that information matters.

Approach A: reward shaping with an information bonus. Keep a single scalar objective and add an intrinsic term:

In this expression:

• $H$: the rollout horizon, or number of decision steps before the episode ends
• $\gamma \in [0,1]$: the discount factor; for $\gamma<1$, it weights near-term reward more heavily than distant reward, while $\gamma=1$ leaves rewards undiscounted
• $R(s_t, a_t)$: the extrinsic reward received for taking action $a_t$ in state $s_t$
• $\beta \ge 0$: the intrinsic weight, controlling how much the agent cares about learning
• $I_t$: the information gain at step $t$, often $I(b_t, a_t)$ when the agent maintains a belief over latent state
• $s_t$: the state at time $t$; if the agent only observes $o_t$, write the expectation under the belief $b_t(s_t)$ rather than under a known state

This is a common approach in curiosity-driven exploration and in some model-based systems. The expectation runs over trajectories generated by the transition dynamics and the policy. The coefficient $\beta$ has units of reward per unit of information, so it also fixes the otherwise arbitrary relative scale.

Approach B: explicitly value the belief. Solve the belief-MDP and let the value function do the work:

where $P(o \mid b, a)$ is the same predictive observation probability as above and $\tau(b, a, o)$ is the Bayesian update.

In this expression:

• $V^*(b)$: the optimal value of belief $b$, i.e., the maximum expected discounted return obtainable from $b$
• $\bar{R}(b, a) = \sum_s b(s)\, R(s, a)$: the expected immediate reward when action $a$ is taken under belief $b$
• $P(o \mid b, a)$: the predictive observation probability, i.e., $\sum_{s'} O(o \mid s', a) \sum_s T(s' \mid s, a)b(s)$
• $\tau(b, a, o)$: the Bayesian update that maps belief $b$, action $a$, and observation $o$ to the posterior belief
• $\gamma \in [0,1]$: the discount factor

In this formulation there is no arbitrary $\beta$: the value of information is whatever it contributes to task reward under the model. It is exact under the stated POMDP assumptions, but often computationally intractable because the belief space and observation tree are large. For the usual infinite-horizon contraction argument, take $\gamma \in [0,1)$; an undiscounted problem needs a finite horizon or additional conditions.

Approach A is a tractable shaped objective, not a general approximation theorem for Approach B. It agrees with task value only under additional assumptions linking information to later reward. A task-irrelevant observation can carry mutual information while leaving every useful decision unchanged. We will return to this mismatch when we discuss limitations.

The structural difference is worth stating plainly. Approach B values information only through posterior-contingent task actions. Approach A values a chosen information target directly. A discounted sum of bonuses can plan over several steps, so it is not necessarily myopic in time; its weakness is that it remains task-agnostic unless the bonus is explicitly decision-aware. It may reward information that arrives too late or concerns distinctions no action uses.

Picture the warehouse robot again. Its belief over the target pallet's identity is a distribution: mostly "pallet A," with some mass on "pallet B." Driving straight to A has high expected reward if the belief is right, but if it is wrong the reward is negative. Rotating the camera is diagnostic because the conditional likelihoods $P(o \mid s, \text{rotate})$ are well separated for the two pallet identities. Whichever label is observed therefore shifts the posterior strongly. That is different from saying that the predictive mixture $P(o \mid b, \text{rotate})$ is already concentrated on the unknown true label. A dual controller may pay the sensing cost and then drive; a greedy controller commits immediately; a pure information seeker can continue probing after further observations no longer help the task.

The example generalizes to any sensor the agent commands. A microscope can be refocused, a hand can be rotated to feel the back of an object, a query can be sent to an oracle, a temperature reading can be repeated. In every case the agent weighs an action's task effect against what it reveals for the next decision. The question is how much an action is expected to reduce uncertainty about the hidden variable, relative to its predictive prior, and how that learning changes the next best decision. A particular observation can still make the posterior more diffuse.

The failure mode of the perpetual student recurs throughout the chapter, so it deserves a name.

Now that we have a way to talk about actions carrying two kinds of value, we can make the "knowledge" part quantitative. Information gain gives us the quantitative handle on how much a candidate action would teach the agent, and the next section defines it, connects it to the belief-MDP recursion, and shows how to compute it in the Gaussian and ensemble approximations.

Information gain is the expected reduction in uncertainty about a specified random variable that results from taking an action and observing its outcome. We will define it in the Shannon sense, derive its connection to the belief-MDP, and then compute it. Standard continuity, symmetry, and composition axioms single out Shannon entropy up to a choice of scale. Other uncertainty and risk measures remain useful under different objectives, so the information target and decision criterion must always be named.

For a discrete belief $b$ over a fixed finite state set, the entropy is

where:

• $b(s)$: the belief probability assigned to state $s$, with $b(s) \ge 0$ and $\sum_s b(s) = 1$
• $\log$: the natural logarithm, so entropy is measured in nats
• the sum runs over all states $s$ in the state space

with the convention $0 \log 0 = 0$. Entropy is nonnegative, zero exactly when the belief is a point mass, and maximized by the uniform distribution. It is measured in nats (natural logs) or bits (base-2 logs); the choice of base rescales entropy and information gain by a constant.

For a dynamic hidden state, the prior used for the next observation is not generally the current belief. After action $a$, the predictive belief over the next state is

The observation-conditioned posterior is $\tau(b,a,o)(s')$. Its entropy is

where $\tau$ is again the Bayesian update. The expected posterior entropy after taking action $a$ is the average over possible observations weighted by their predictive probability:

This expression averages over what the observation could turn out to be. On the right, the sum weights each possible posterior entropy $H(\tau(b, a, o))$ by the probability $P(o \mid b, a)$ that the corresponding observation occurs. Once you have seen the observation, the posterior is deterministic, so there is no residual averaging inside a given branch; the averaging is entirely over which branch you land in.

The information gain about the next state from action $a$ is the mutual information between $S'$ and $O$, conditional on the current belief and action:

where:

• $H(b_a^-)$: the entropy of the action-conditioned predictive prior over $S'$
• $P(o \mid b, a)$: the predictive probability of observing $o$ after taking action $a$ from belief $b$
• $H(\tau(b, a, o))$: the entropy of the posterior over $S'$ after observing $o$
• $\tau(b, a, o)$: the Bayesian posterior update
• $I_{S'}(b, a)$: the expected information gain, in nats, about the named target $S'$

For a static hidden target or parameter $X$, action does not transition $X$, so the predictive prior equals the current prior. The worked example later uses exactly this special case: $X$ is the identity of the rewarded target and $b_a^-=b$.

This quantity has three equivalent interpretations that are worth stating, because different literatures emphasize different ones:

• Entropy reduction: how much, on average, the observation reduces uncertainty about the named latent variable.
• Mutual information: $I(S';O\mid b,a)$, the dependence between predictive next state and observation for the chosen action.
• Expected posterior-to-predictive-prior KL: $I_{S'}(b,a)=\mathbb{E}_{o}[D_{\mathrm{KL}}(\tau(b,a,o)\|b_a^-)]$.

The third form is useful for Bayesian-surprise methods. Other curiosity bonuses introduced later are only surrogates for this quantity.

Write the expected KL from predictive prior to posterior. Expanding the KL gives

Substitute $\tau(b,a,o)(s')=O(o\mid s',a)b_a^-(s')/P(o\mid b,a)$ and rearrange:

This is the standard mutual-information identity for the joint distribution induced by the predictive prior and observation channel. The prior and posterior must concern the same random variable; comparing a posterior over $S'$ with a pre-action belief over $S$ is not meaningful in general.

Reading the algebra backward is useful, with one qualification. Maximizing expected posterior-to-predictive-prior KL maximizes information gain about the named variable. Minimizing expected posterior entropy is equivalent only when predictive-prior entropy is fixed across the actions being compared. Bayesian-surprise methods use this identity directly; count bonuses, prediction error, RND, and ensemble disagreement are different surrogates and need not estimate the same quantity.

A useful sanity property: $I_{S'}(b, a) \ge 0$ for every belief and action, with equality if and only if $O$ and $S'$ are conditionally independent for that $(b,a)$. This follows directly from $D_{\mathrm{KL}} \ge 0$. A negative numerical estimate therefore reflects approximation, sampling, or floating-point error rather than negative exact mutual information.

Information gain is also invariant to relabeling of the observations: it is built from posterior entropies, which do not depend on the symbols assigned to those posteriors. If you permute the observation symbols and the observation model consistently, information gain does not move. An action that reliably reveals "not A" is just as informative as one that reliably reveals "A": what matters is how sharply the observation is correlated with the latent state, not which symbol encodes it.

Look back at the belief-MDP recursion. The term $P(o \mid b, a)$ is the branch probability, and the downstream value $V^*(\tau(b,a,o))$ depends on the posterior. Information gain measures the expected posterior-to-predictive-prior KL induced by these branches. The weight $\beta$ converts information into objective units, but it cannot by itself determine whether the information changes a useful decision.

Not all information is equally valuable. The quantity relevant to control is not entropy in general but entropy that affects the optimal action. If two states are equally good under every action, distinguishing them has no value. This is the "value of information" perspective. It motivates a refinement of plain entropy.

The inner maximum chooses the best task decision after seeing $o$, whereas entropy-based information gain does not ask whether the posterior changes a useful choice. Resolving an irrelevant distinction changes the belief but leaves the maximum unchanged. This mismatch reappears in curiosity, where surprise-driven objectives can chase irrelevant novelty, and in safety, where useful probes may be inadmissible.

For continuous latent states, the exact belief is rarely available. Differential entropy depends on coordinates and may be negative or divergent; mutual information, when well defined, is invariant under smooth invertible reparameterizations. Practical systems therefore estimate mutual information directly or use a calibrated parametric or ensemble surrogate.

For a nonsingular Gaussian belief $p(s) = \mathcal{N}(s; \mu, \Sigma)$ over a $d$-dimensional continuous latent state, with positive-definite covariance $\Sigma$, the differential entropy is

where:

• $d$: the dimension of the latent state
• $\mu$: the mean of the Gaussian belief
• $\Sigma$: the covariance matrix encoding the belief's uncertainty
• $\det \Sigma$: the determinant of the covariance; $\sqrt{\det\Sigma}$ is proportional to the volume scale of a fixed Mahalanobis-radius ellipsoid
• $e$: Euler's number, arising from the maximum-entropy property of the Gaussian

Three implementations recur throughout this book:

• Ensemble disagreement: train $K$ diverse models and use their predictive spread as an epistemic proxy. That interpretation requires posterior-like diversity, calibration, and a separate representation of aleatoric output noise. Shared data alone does not make disagreement epistemic. PETS, for example, combines ensemble variation with probabilistic member outputs.
• Parametric differential entropy: for nonsingular Gaussian predictive and posterior beliefs, $$I=\tfrac12\left[\log\det\Sigma_{\text{prior}}-\mathbb{E}_o\log\det\Sigma_{\text{post}}(o)\right].$$ In a linear-Gaussian update the posterior covariance is independent of the realized observation, so this simplifies to one half the log-determinant ratio.
• Variational information estimates: optimize a learned lower or upper bound on the expected log density ratio, often through a conditional density or contrastive classifier. A raw target-class probability is not itself mutual information.

These are useful approximations, but the executable example below deliberately uses an exact two-state categorical belief so every posterior and information value can be checked directly. First, we sharpen the notion of curiosity, which is the motivational layer on top of information gain.

Information gain is a property of an action given a belief. Curiosity is an agent-level disposition: a tendency to choose such actions, especially when extrinsic reward is sparse or absent. In reinforcement learning, we implement curiosity by adding an intrinsic reward $r^{\text{int}}_t$ to the environment's extrinsic reward $r^{\text{ext}}_t$ and optimizing the sum (or a discounted combination). The intrinsic term is a hand-designed or learned signal that rewards the agent for experiencing informative or novel states.

A useful mental model treats the intrinsic reward as a subsidy. The environment pays the agent for task progress, but the task is often so sparse that the agent never finds the first payment. The intrinsic reward is a short-term subsidy that keeps the agent searching. The subsidy is not free: it changes the incentives the agent sees, and if it is paid forever the agent may spend its whole life collecting it. The engineering problem of intrinsic motivation is to design a subsidy that helps the agent find the real reward and then gets out of the way.

Two complementary motivations drive intrinsic rewards.

1. Statistical: In sparse-reward environments, few sampled trajectories contain nonzero returns, which makes credit assignment and gradient estimation difficult. An intrinsic reward supplies a denser learning signal.
2. Computational hypothesis: organisms do explore without immediate task reward, and computational accounts model curiosity as learning or compression progress. This is a hypothesis about a useful intrinsic objective, not an established evolutionary explanation.

World models make the statistical case concrete: coverage of decision-relevant state-action regions is one requirement for reliable planning, alongside identifiability, a suitable model class, and robust decision making under residual error. Intrinsic reward can improve the data used by the model, not just help the policy stumble onto task reward. This connects to [Online, Offline, and Continual Learning].

Intrinsic rewards differ in what they reward. The main families are:

• Count-based: reward $1 / \sqrt{N(s,a)}$ or a pseudo-count analogue. Density surprisal $-\log \hat p(s)$ is related but distinct. Optimistic model-based algorithms such as R-MAX also use visitation thresholds, but they do not implement this inverse-count reward directly.
• Prediction-error (surprise): reward $-\log \hat{P}(s'_t \mid s_t, a_t)$. This is a surprise bonus, not information gain in general: irreducible stochasticity can keep it high after there is nothing left to learn, producing the noisy-TV problem.
• Ensemble disagreement: reward spread across diverse model predictions. With calibrated members and aleatoric uncertainty modeled separately, disagreement can serve as an epistemic proxy; those conditions are not automatic.
• Information-theoretic / Bayesian surprise: reward the realized posterior-to-prior KL divergence after an observation. Its expectation over observations is mutual information; the realized quantity need not equal the expected entropy reduction for a particular outcome.
• Empowerment: maximize channel capacity from action sequences to future states, commonly written $\max_{p(a_{t:t+k-1}\mid s_t)} I(A_{t:t+k-1};S_{t+k}\mid s_t)$. This asks how many distinguishable future outcomes the agent can control and connects to the skills in [Hierarchical, Symbolic, Language, and Multi-Agent Planning].
• Novelty in learned latent space: reward the novelty of a state's embedding under an encoder, which avoids counting in raw dimensions.

A useful way to organize this taxonomy is by what is being predicted: the agent can be curious about states (visit novel states), about transitions (learn the dynamics), about parameters (identify the system, cf. [System Identification]), or about its own abilities (discover skills). These reward different things and can be combined.

Suppose you want an agent to learn a manufacturing line that occasionally jams. A count bonus tends to spread visits, a prediction-error bonus may overvalue the irreducibly random jam, and a calibrated ensemble may focus on regions where plausible dynamics models disagree. An information-theoretic objective instead rewards observations that update a named belief about the line. These are intended tendencies, not deterministic outcomes: representation, calibration, optimization, and bonus scale can reverse them.

The standard formalization of the tradeoff is the regret of a bandit or MDP. For a multi-armed bandit with $K$ arms and mean rewards $\mu_i$, let $\mu^* = \max_i \mu_i$ be the best-arm mean. The regret after $T$ pulls is

where:

• $K$: the number of arms
• $\mu_i$: the unknown mean reward of arm $i$
• $\mu^* = \max_i \mu_i$: the mean reward of the best arm
• $a_t \in \{1, \dots, K\}$: the arm pulled at time step $t$
• $\mu_{a_t}$: the mean reward of the arm chosen at step $t$
• $T$: the total number of pulls, or horizon

This quantity is pseudo-regret relative to always pulling the arm with the highest mean reward, not realized hindsight regret. Sublinear pseudo-regret means average per-step regret goes to zero as $T$ grows. A policy that permanently commits to a suboptimal arm with gap $\Delta>0$ incurs $\Delta T$ pseudo-regret.

• $\varepsilon$-greedy: act greedily with probability $1-\varepsilon$, randomly otherwise. With fixed $\varepsilon$, exploration neither vanishes nor responds to arm-specific uncertainty. A decaying schedule can use time, but remains externally specified unless it is tied to uncertainty.
• Optimism (UCB1): after pulling every arm at least once, choose $\arg\max_i [\hat{\mu}_i(t)+\sqrt{2\log t/N_i(t)}]$. For independent stationary arm rewards normalized to $[0,1]$, UCB1 has logarithmic instance-dependent expected regret. For rewards in a known interval $[a,b]$, normalize them or scale the confidence radius by $b-a$.
• Thompson sampling: sample a reward estimate from the posterior and act greedily on the sample. This gracefully balances exploration and exploitation because widely uncertain arms get sampled high some of the time.
• Information-directed sampling (IDS): choose an action distribution that minimizes squared expected one-step regret divided by mutual information about the optimal action. Information-ratio analyses yield Bayesian regret bounds for broad model classes; the statement is not an unconditional near-optimality guarantee.

For MDPs, related ideas use state-dependent counting, ensembles, uncertainty propagation, conservative penalties, or short synthetic rollouts. These mechanisms are not interchangeable: PETS propagates probabilistic ensemble predictions while optimizing task reward, whereas MBPO limits model bias with short model rollouts. Part VIII develops those distinctions.

For standard policy-gradient, actor-critic, or value-based learning, an intrinsic reward only needs to be computable as a scalar; the optimizer does not differentiate through the reward generator or environment. Differentiability is additionally useful when gradients are propagated through a learned model or planner. Two standard constructions are:

RND-style (random network distillation). Fix a randomly initialized network $f_\theta$ and train a predictor $g_\phi$ to match its output on visited states. The intrinsic reward is $\|g_\phi(s) - f_\theta(s)\|^2$. Novel states often have higher prediction error when the predictor does not generalize to them. The mismatch between the fixed random target and the learned predictor is a stand-in for surprise.

Latent-space disagreement. Train a diverse ensemble $\{h_k\}$ of encoders or dynamics models. The intrinsic reward is the variance or Jensen-Shannon divergence across comparable member predictions. Its epistemic interpretation depends on diversity, calibration, and separate modeling of observation noise.

Both compose with standard policy optimizers. A practical failure is a persistent prediction-error bonus: the agent finds observations that remain hard for the predictor for reasons unrelated to useful novelty and farms them. The residual may be driven by background clutter, sensor noise, limited predictor capacity, or nonstationarity rather than by reducible uncertainty.

Consider a toy robot in a room with a light switch. Pressing it may initially be informative about the lighting model. As evidence identifies the causal link, Bayesian information gain about that parameter falls. An exact count bonus also decays with visits, but its decay tracks visitation rather than whether the causal parameter has been learned. A prediction-error agent can still be fooled by random flicker because irreducible surprise remains high.

The example shows why novelty differs from information. Count novelty decreases as a state is revisited; information gain decreases as uncertainty about a specified latent variable is resolved. The two can fall at different rates and can generalize differently across representations. Decision value adds a third test: even information about the named variable may not change the task action.

The intrinsic weight $\beta$ is a first-class hyperparameter. Too small and curiosity may not change behavior; too large and the policy may optimize the proxy instead of the task. One option is to anneal $\beta$; others normalize the two returns or learn separate value heads. A multiplicative form such as $r^{\text{ext}}(1+r^{\text{int}})$ is task-dependent: it erases exploration wherever $r^{\text{ext}}=0$ and behaves poorly when rewards change sign, so it provides no general protection against intrinsic-reward domination.

We now have an information objective and a motivation to pursue it. The remaining piece is making sure the pursuit is safe.

Exploration is the deliberate acquisition of information, and information acquisition can hurt. A house-cleaning robot that explores a staircase learns about the stairs; it may also fall. An autonomous car that explores aggressive steering learns about the tire model; it may also leave the road. Safety-constrained exploration asks how an agent can learn the cheapest possible lessons about the hardest constraints. The theoretical guarantees in this area are thinner than elsewhere in the chapter, so we distinguish carefully among approaches and their assumptions.

A standard formalism is the constrained MDP (CMDP). It adds a set of cost functions $C_i(s, a)$, each with a budget $c_i$:

with $\gamma\in[0,1)$ for bounded infinite-horizon returns.

Costs encode quantities to constrain, such as collision indicators, energy, torque, or risk exposure. An expected discounted-cost budget can permit rare costly trajectories while satisfying the expectation. That criterion is distinct from cumulative violation regret, a chance constraint, an almost-sure constraint, or robust pathwise invariance. None implies another without a specific theorem and assumptions, so a safety claim must name both the event being bounded and the probability space.

A hard constraint is intended to hold at every step rather than only on average. A controlled-invariant safe set $\mathcal{S}_{\text{safe}}$ is one for which there exists an admissible feedback policy that keeps the trajectory inside for all disturbances covered by the model. Requiring the set to remain safe under every admissible control is the stronger notion of strong invariance; robustness instead refers to preserving the relevant invariance property under the specified disturbances or model uncertainty. Two canonical tools are:

• Control barrier functions (CBFs): a scalar certificate $h$ whose superlevel set $\{s:h(s)\ge 0\}$ defines the candidate safe set. Under regularity assumptions, if a feasible control satisfies a condition such as $\dot h(s,a)\ge-\alpha(h(s))$ for an appropriate class-$\mathcal K$ function $\alpha$, that set can be forward invariant for the modeled dynamics. A CBF need not be a signed distance. With state uncertainty, one conservative option enforces the condition over a confidence set; chance-constrained and belief-space barrier designs encode uncertainty differently.
• Shielding / safety filters: take the exploratory action $a_{\text{exp}}$ and test its feasibility (against a model, a reachability computation, or a hard-set controller). A filter may accept, repair, or replace the proposal with an admissible action; projection is one possible repair, not the definition. This interlock permits exploration within its modeled constraints when a feasible safe response exists. For model-based agents the filter is itself model-based, so its guarantee is only as good as the model (a model mismatch failure mode we discuss in the reliability and safety chapters).

The safe exploration literature distinguishes several non-equivalent guarantee types:

• Expected-budget safety constrains expected cumulative cost and may still permit costly trajectories.
• Violation-regret guarantees bound cumulative excess or the number of violations under a particular online-learning theorem.
• High-probability or chance constraints bound a stated event with probability at least $1-\delta$. Depending on the theorem, the probability may cover estimation data, process and observation noise, policy randomness, or their joint law.
• Pathwise or invariant-set guarantees keep every modeled trajectory in a safe set under specified disturbances and a feasible controller.

SafeOpt belongs to the high-probability category: under its Gaussian-process regularity, confidence, and initial-safe-set assumptions, it restricts evaluations to points certified safe on the confidence event. It is not a sublinear-violations method.

These guarantees rest on method-specific assumptions, for example a safe seed set, regularity, calibrated confidence bounds, a sound abstraction, feasible safe controls, bounded disturbances, or a certified backup controller. A CBF-based controller is safe only if its barrier conditions hold for the true dynamics. Near a boundary, model mismatch can make the filter satisfy its modeled inequality while the physical system crosses the boundary. This is why guarantees in learned-model settings must state their assumptions next to the guarantee.

Safety and exploration can conflict in a structural way. Informative actions can lie near a known-safe boundary when that boundary also marks the edge of the data-supported region. This alignment is common in some safe-expansion problems but is not guaranteed. When it does occur, a small model error near the boundary can become a violation. Three ways practitioners resolve the tension are:

1. Explore from a known-safe baseline. A certified controller executes or recovers while a learned policy suggests probes. The permission rule must preserve enough recoverability margin; offline conservative value estimation is not itself a certified safety fallback.
2. Explore in simulation, deploy conservatively. This is common in some robotics applications, but simulator violations are not literally free: they still consume compute and can teach policies that exploit simulator errors. The main deployment risk is the sim-to-real gap.
3. Optimize a risk-sensitive surrogate. Soft penalties or a tail-risk objective such as CVaR can discourage hazardous probes. They change preferences but do not enforce a hard safety constraint.

Since information gain is computed under a probability model, and the consequences of probing are random, it is natural to make the exploration objective risk-sensitive. The standard device is the conditional value at risk (CVaR): for a random cost $Z$ and risk level $\alpha \in (0,1)$,

where $(x)_+=\max(x,0)$. For a continuous loss distribution with no atom at its $\alpha$-quantile, this equals the conditional mean above $\mathrm{VaR}_\alpha$; the optimization form remains valid for discrete distributions with mass at the quantile. A CVaR penalty or constraint controls average upper-tail cost. It can still choose a risky probe if the permitted bound or reward tradeoff favors it, so it does not guarantee zero tail exposure. Unlike variance, upper-tail CVaR targets large costs rather than deviations in both directions.

Classical optimism says "act as if the world is as good as it plausibly could be." For safety constraints, the analogue is the reverse: "act as if the world is as bad as it plausibly could be." These two principles routinely appear in the same algorithm, one governing reward and one governing cost. Mixing them up is a classic bug. Keep the sign conventions explicit in code: reward bonuses are added, cost bonuses are subtracted or bounded.

We now have all the conceptual ingredients: information gain as the knowledge objective, curiosity as the motivation to pursue it, and constraints as the limit on how far to go. The next section builds a small end-to-end system.

We will construct a small partially observable gridworld in which the agent must reach one of several targets while its identity is uncertain, and it can optionally take an inspect action that reveals a noisy label. Then we will compare three policies: greedy reward, a thresholded curiosity heuristic, and a dual controller that trades task value against an optional information bonus. The worked example is deliberately tiny so we can compute the exact belief and information gain and see the tradeoff numerically.

The world is a $5 \times 5$ grid. There are two candidate targets, A and B, placed at fixed cells. The agent knows the layout but not which target is rewarded; its prior is $P(\text{target}=A)=0.7$. It can move or inspect. An inspection costs $0.18$ reward units and reports the true binary label with probability $0.8$ and the other label with probability $0.2$. Moving onto a target yields $+1$ if it is rewarded and $-1$ otherwise; movement itself has zero cost. The episode ends at either target. The implementation uses fixed paired episode conditions so policies face the same target identities and observation noise.

The next cells implement the environment, the exact Bayesian update, and the information gain.

All four unfiltered proposals enter the hazard; all four filtered actions avoid it. This establishes only a one-step invariant for the deterministic grid and known hazard. It does not establish route completion, robustness to unknown hazards, or safety under model mismatch. A formal shield needs a sound transition abstraction or model, a specification, and a proof that a safe action remains available over the relevant horizon.

The key parameters for the gridworld exploration example are:

• INSPECT_ACC: The probability that inspecting reveals the true target label. For this symmetric channel, values farther above chance, from $0.5$ toward $1.0$, make inspection more informative and generally reduce the number of probes needed.
• INSPECT_COST: The reward cost charged for each observation. It determines when the one-probe task value exceeds immediate commitment.
• beta: The additional reward-per-nat weight on raw information. In this example small values leave the decision-aware policy unchanged, while large values cause avoidable extra probes.
• threshold: The information-gain cutoff used by the curious policy. Lower thresholds cause inspection on weaker evidence.
• HAZARD: The known forbidden cell used by the one-step action filter. It is not a learned or probabilistic hazard model.

The value-of-information gap is a practical liability. Raw information gain and prediction-error curiosity can be misaligned with task value. An agent may resolve an unreachable target's identity or an irrelevant visual detail while missing uncertainty that changes the action. Belief-MDP planning and IDS-style information ratios make the decision link explicit, but they are usually more expensive. Practical systems therefore treat intrinsic reward as a surrogate and evaluate it on the extrinsic task.

Persistent prediction error and noisy TV are related proxy failures with different mechanisms. Noisy TV is driven by aleatoric unpredictability. Persistent RND or model error can also come from limited capacity, optimization, nonstationarity, or nuisance features. Remedies therefore differ: inverse-dynamics features can suppress uncontrollable pixels, explicit aleatoric modeling can separate noise from model uncertainty, and learning-progress rewards can pay only when error decreases. Ensemble disagreement is only an epistemic proxy and must be calibrated rather than assumed correct.

Bias from passive data. As covered in [Interaction Data and Passive Observation], intrinsic rewards depend on the data distribution, and an agent that only ever collects the data its current policy induces has a self-reinforcing sampling bias. It will fail to discover regions it never visits and cannot assess the value of information it never obtains. Off-policy and exploratory priors help, but they do not eliminate the issue. A model that is overconfident in unvisited regions also makes probing look uninformative, which further suppresses exploration. This is the exploration collapse failure mode: the agent becomes certain it has nothing left to learn while being wrong.

Safety guarantees are conditional on their assumptions. SafeOpt, CBFs, shields, chance constraints, and recovery policies use different assumptions: safe seeds, regularity, sound abstractions, calibrated confidence sets, feasible controls, disturbance models, or certified backups. Model-based filters inherit model mismatch, especially near poorly observed boundaries. In a high-stakes setting, state the exact guarantee and probability space, preserve a validated fallback where available, and do not treat a learned score as a certificate.

Combinatorial action spaces and long-horizon exploration. One-step information gain is easy to state but useful observations may require several coordinated actions. Belief-space planning represents that horizon but faces branching and dimensionality; intrinsic bonuses compose with scalable policy learners but solve neither dimensionality nor long-horizon credit assignment by themselves. Hierarchical methods in [Hierarchical, Symbolic, Language, and Multi-Agent Planning] provide another way to structure the search.

Evaluation is subtle. Intrinsic reward changes the policy and the data distribution, so its training value is not task performance. Evaluate extrinsic return, safety events, uncertainty calibration, and held-out tasks against an exploration-free baseline. Decision-relevant state-action coverage can be a useful diagnostic, but it does not guarantee generalization. Seed sensitivity, tuning budgets, baselines, normalization, distribution shift, and statistical power can all change a comparison.

Active perception and dual control offer a common formal lens for closed-loop sensing, system identification, viewpoint selection, sparse-reward exploration, and experiment design. Curiosity has enabled important exploration results, while many model-based RL systems instead rely on random collection, MPC, ensembles, or short model rollouts without an intrinsic bonus. Reward maximization and knowledge acquisition are different objectives; making them compatible requires an explicit decision model, scale, and safety criterion.

Active perception and dual control study actions whose physical and informational consequences are optimized jointly. The key takeaways are that information gain quantifies the informational consequence, curiosity motivates pursuing it, and safety constrains how far the pursuit can go.

• Belief-MDPs make knowledge a state. Under a known POMDP and exact filtering, the posterior is a sufficient statistic, so information affects task value through posterior-contingent future actions.
• The dual effect requires control-dependent information. Standard LQG separation is a special case. Nonlinearity, multiplicative control, or parameter uncertainty can reintroduce probing when actions change what later observations can reveal.
• Information gain names its target. For next-state information it equals $H(b_a^-)-\mathbb{E}_oH(\tau)$ and the expected $D_{\mathrm{KL}}(\tau\|b_a^-)$. In Gaussian approximations it uses an expected posterior log determinant; a single log-determinant ratio needs observation-independent posterior covariance.
• Information gain is not value of information. The first is measured in nats; the second is measured in reward. Information can be positive while task value is zero, and no numerical ordering follows without utility-scale assumptions.
• Curiosity is the motivation to pursue information, often implemented as an intrinsic reward on top of extrinsic reward, with families spanning count-based, prediction-error, ensemble-disagreement, information-theoretic, empowerment, and latent-novelty bonuses. Their proxy-dependent failures include noisy-TV attraction, nuisance novelty, miscalibrated disagreement, and exploration collapse.
• Safety criteria are not interchangeable. Expected CMDP budgets, violation regret, chance constraints, invariant sets, CVaR, and shields make different claims over different randomness. Every guarantee is conditional on its method's model, feasibility, confidence, or fallback assumptions.
• The tradeoff is context-dependent. The optimal quantity of exploration changes with the belief, the horizon, the cost of error, and the safety budget. The dual weight and the exploration strategy must therefore be tuned per problem rather than chosen once.

We have now covered the inward-facing half of the planning-and-agency part: how an agent can act on its own uncertainty to gather information, and how safety criteria restrict that choice. Next, in [Hierarchical, Symbolic, Language, and Multi-Agent Planning], we turn outward and consider agents that plan at multiple levels of abstraction, coordinate with other agents, and use language as a planning substrate. The same belief and information primitives reappear there across longer temporal and social structures.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about active perception, dual control, and exploration.