Active Perception, Dual Control, and Exploration › Actions That Improve Both Knowledge and Reward

Before we can say an action "improves knowledge," we need a quantity that measures knowledge and a way to say an action changes it. The prior chapter on Bayesian Filtering and Belief States already gave us the first: the agent maintains a belief $b(s)$ over the state, and actions plus observations update that belief. The quantity we will use to compare beliefs is the entropy of the belief. But to make the reward and knowledge objectives comparable, we need to frame the whole problem as a single decision problem, and for that the belief must be part of the state the planner reasons over.

This section builds that framing in three steps. First we review the belief-MDP, the object that turns "knowledge" into something a planner can value like any other state feature. Then we introduce Feldbaum's dual effect, the phenomenon that makes probing a control problem rather than a bookkeeping nuisance, and we are careful about exactly when the effect is present. Finally we write down two ways to tuck information into the objective, one tractable and approximate, one exact but intractable, and we are explicit about the gap between them.

Recall from [Markov and Partially Observable Decision Processes] that a partially observable Markov decision process (POMDP) has states, actions, observations, a transition model $T(s' \mid s, a)$, an observation model $O(o \mid s', a)$, and a reward $R(s, a)$. When the true state is not known, the agent acts on its belief $b$, the posterior distribution over the state given the history. The key structural fact is that, under the assumption that the underlying state is Markov, the belief itself is a sufficient statistic for the history and evolves in a Markov way. Formally, the belief $b$ plays the role of a state in a new, fully observable process called the belief-MDP (also written belief-state MDP or B-MDP). Its transition is deterministic given an observation. Bayes' rule for the belief update is:

The denominator $P(o \mid b, a)$ is the predictive observation probability;

This denominator is what makes the belief-MDP legitimate: it is the normalizer that keeps the posterior a valid distribution, and as we will see, it is also the engine of information gain.

where:

• $b(s)$: the current belief probability assigned to state $s$ before action $a$ is taken
• $T(s' \mid s, a)$: the transition probability of moving from state $s$ to state $s'$ under action $a$
• $O(o \mid s', a)$: the observation probability of seeing $o$ after landing in $s'$ under action $a$
• $b'(s')$: the posterior belief probability assigned to state $s'$ after seeing observation $o$
• $P(o \mid b, a)$: the predictive probability of observation $o$ under belief $b$ and action $a$, which normalizes the posterior

The belief $b(s)$ is a probability, so it is nonnegative and sums to one; it encodes everything the agent knows about the state given its history. The transition $T$ describes how the world moves in the absence of observations, and the observation model $O$ describes how the world reveals itself. The two combine, through Bayes' rule, into the posterior $b'(s')$, which is what the agent believes after seeing $o$. Because $b'$ is a deterministic function of $(b, a, o)$ and $o$ is random, the belief-MDP is a fully observable stochastic process whose "state" is the belief. In other words, the randomness in the belief-MDP comes entirely from the observation $o$; once you fix the belief, the action, and the observation, the next belief is fixed. This property is what allows us to speak of the belief as a state in the ordinary MDP sense.

\newpage

The reason to build this object is that it lets us talk about "value of information" through ordinary continuation value: an action can move the belief to a region where later choices earn more reward. The belief-MDP is an exact reformulation when the model is known and the belief is maintained exactly. In general, the unrestricted space of distributions over a continuous state is infinite-dimensional. Special closed families, such as Gaussian beliefs in a linear-Gaussian model, have finite-dimensional sufficient parameters; outside such cases, practical methods approximate the belief or its value.

Feldbaum's dual control is the observation that in a stochastic control problem the action $a_t$ affects both the future distribution of states and the future distribution of beliefs about those states. These are the two "effects." The first is the ordinary control effect: $a_t$ changes the next-state distribution $p(s_{t+1} \mid b_t, a_t) = \sum_s b_t(s) T(s_{t+1} \mid s, a_t)$. The second is the probing and learning effect: different actions produce different observation likelihoods $P(o \mid b, a)$, so they lead to different posteriors $b_{t+1} = \tau(b_t, a_t, o_t)$ and different future uncertainty.

The probing effect is not always present. In standard linear-quadratic-Gaussian (LQG) control, the dynamics and observations are known and linear, process and observation noise are additive Gaussian and independent of control, and the objective is an expected quadratic cost without control-dependent sensing or hard constraints. Under those assumptions, the separation principle holds: the optimal controller feeds the Kalman conditional mean into the full-state linear-quadratic controller. The mean is also the MAP estimate in the Gaussian case. The covariance follows a Riccati recursion determined by the model and noise covariances, not by the realized control inputs, and the Kalman gains are computed from that covariance. The controller therefore has no incentive to probe merely to improve estimation.

The separation property is special, not generic. A dual effect can arise when actions alter the information available to later decisions, for example when:

• Nonlinear dynamics or state-dependent sensors make posterior uncertainty depend on where actions move the system.
• Control enters the state or observation channel multiplicatively, making the signal quality action-dependent.
• Unknown parameters are identifiable only under particular inputs, so controls determine how quickly the system can be learned.

None of these features is sufficient by itself. A fully observed nonlinear system has no hidden state to probe, and an unknown but unidentifiable parameter cannot be learned by any action. Reward and safety constraints determine whether an informative probe is worth taking or admissible; they do not create the information channel.

For the rest of this chapter we focus on regimes in which actions change decision-relevant uncertainty, as many online, partially observed world-model agents do.

The practical consequence is conditional rather than universal. Many online world-model agents do operate in a dual-effect regime because their actions determine future observations and parameter updates. A fixed learned model used in a fully observed task does not: learning provenance alone does not create a probing effect. The defining test is whether a current action changes uncertainty that matters to later decisions.

There are two broad ways to write down the objective once we accept that information matters.

Approach A: reward shaping with an information bonus. Keep a single scalar objective and add an intrinsic term:

In this expression:

• $H$: the rollout horizon, or number of decision steps before the episode ends
• $\gamma \in [0,1]$: the discount factor; for $\gamma<1$, it weights near-term reward more heavily than distant reward, while $\gamma=1$ leaves rewards undiscounted
• $R(s_t, a_t)$: the extrinsic reward received for taking action $a_t$ in state $s_t$
• $\beta \ge 0$: the intrinsic weight, controlling how much the agent cares about learning
• $I_t$: the information gain at step $t$, often $I(b_t, a_t)$ when the agent maintains a belief over latent state
• $s_t$: the state at time $t$; if the agent only observes $o_t$, write the expectation under the belief $b_t(s_t)$ rather than under a known state

This is a common approach in curiosity-driven exploration and in some model-based systems. The expectation runs over trajectories generated by the transition dynamics and the policy. The coefficient $\beta$ has units of reward per unit of information, so it also fixes the otherwise arbitrary relative scale.

Approach B: explicitly value the belief. Solve the belief-MDP and let the value function do the work:

where $P(o \mid b, a)$ is the same predictive observation probability as above and $\tau(b, a, o)$ is the Bayesian update.

In this expression:

• $V^*(b)$: the optimal value of belief $b$, i.e., the maximum expected discounted return obtainable from $b$
• $\bar{R}(b, a) = \sum_s b(s)\, R(s, a)$: the expected immediate reward when action $a$ is taken under belief $b$
• $P(o \mid b, a)$: the predictive observation probability, i.e., $\sum_{s'} O(o \mid s', a) \sum_s T(s' \mid s, a)b(s)$
• $\tau(b, a, o)$: the Bayesian update that maps belief $b$, action $a$, and observation $o$ to the posterior belief
• $\gamma \in [0,1]$: the discount factor

In this formulation there is no arbitrary $\beta$: the value of information is whatever it contributes to task reward under the model. It is exact under the stated POMDP assumptions, but often computationally intractable because the belief space and observation tree are large. For the usual infinite-horizon contraction argument, take $\gamma \in [0,1)$; an undiscounted problem needs a finite horizon or additional conditions.

Approach A is a tractable shaped objective, not a general approximation theorem for Approach B. It agrees with task value only under additional assumptions linking information to later reward. A task-irrelevant observation can carry mutual information while leaving every useful decision unchanged. We will return to this mismatch when we discuss limitations.

The structural difference is worth stating plainly. Approach B values information only through posterior-contingent task actions. Approach A values a chosen information target directly. A discounted sum of bonuses can plan over several steps, so it is not necessarily myopic in time; its weakness is that it remains task-agnostic unless the bonus is explicitly decision-aware. It may reward information that arrives too late or concerns distinctions no action uses.

Picture the warehouse robot again. Its belief over the target pallet's identity is a distribution: mostly "pallet A," with some mass on "pallet B." Driving straight to A has high expected reward if the belief is right, but if it is wrong the reward is negative. Rotating the camera is diagnostic because the conditional likelihoods $P(o \mid s, \text{rotate})$ are well separated for the two pallet identities. Whichever label is observed therefore shifts the posterior strongly. That is different from saying that the predictive mixture $P(o \mid b, \text{rotate})$ is already concentrated on the unknown true label. A dual controller may pay the sensing cost and then drive; a greedy controller commits immediately; a pure information seeker can continue probing after further observations no longer help the task.

The example generalizes to any sensor the agent commands. A microscope can be refocused, a hand can be rotated to feel the back of an object, a query can be sent to an oracle, a temperature reading can be repeated. In every case the agent weighs an action's task effect against what it reveals for the next decision. The question is how much an action is expected to reduce uncertainty about the hidden variable, relative to its predictive prior, and how that learning changes the next best decision. A particular observation can still make the posterior more diffuse.

The failure mode of the perpetual student recurs throughout the chapter, so it deserves a name.

Now that we have a way to talk about actions carrying two kinds of value, we can make the "knowledge" part quantitative. Information gain gives us the quantitative handle on how much a candidate action would teach the agent, and the next section defines it, connects it to the belief-MDP recursion, and shows how to compute it in the Gaussian and ensemble approximations.