Active Perception, Dual Control, and Exploration › Active Perception, Dual Control, and Exploration

A robot at a warehouse intersection has two equally reachable pallets in front of it. It has never visited this aisle before. Every sensor reading so far has been coarse and noisy. The pallet labels are unreadable until the robot rotates its camera and closes the distance, and the aisle layout is only half-known. If the robot simply drives to whichever pallet looks closer and picks it up, it sometimes grabs the wrong item and pays a penalty. If it instead spends a few seconds rotating and peeking down the aisle, it uses battery and time but learns which pallet holds the target. Neither pure exploitation (act on what you believe right now) nor pure exploration (wander until the world is known) is obviously correct. The right move depends on how much a little information would change the agent's optimal action.

That tension is the subject of this chapter. So far in Part VII: Planning and Agency, we have treated planning as a problem of choosing actions that maximize predicted future reward. Search and Belief-Space Planning already made uncertainty part of the planning state, and Policies Learned in Imagination trained a policy inside a learned model. Here we focus on the next question: when should the agent choose an action partly because of what that action will reveal?

The central idea is dual control, introduced by Feldbaum in 1960: a control action can serve two purposes at once. It can steer the system toward reward and it can probe the system to reduce uncertainty about its state or parameters. An action can therefore have both control value and information value; whether it is preferable depends on the belief, horizon, cost of delay, and later decisions that can use the information. Active perception is a closely related setting in which the agent controls how it senses---by moving a camera, refocusing, touching, or querying---sometimes jointly with task actions.

A treatment of world models devotes a chapter to this topic because a learned world model is only as trustworthy as the data used to train it, and the data an agent collects is shaped by the actions it chooses. An agent that never probes can remain accurate only along its visited state-action tube and may plan confidently outside its data support unless pretraining, off-policy data, model structure, or calibrated uncertainty provides broader coverage. The feedback loop between acting, learning, and planning is the reason exploration cannot always be treated as a preprocessing step. Deciding what to do next and deciding what to learn next are often coupled decisions.

By the end of this chapter you will be able to

• Reason about actions as carriers of both reward and information, and recognize when these two roles reinforce each other and when they conflict.
• Define and compute information gain in the standard Shannon sense, and connect it to uncertainty reduction through the belief-MDP and the mutual-information identity.
• Distinguish information-seeking (intrinsic) objectives from extrinsic reward, understand why the combination is delicate, and identify the characteristic failure modes of each intrinsic-reward family.
• Explore in the presence of safety constraints, where not every informative action is admissible, and state the assumptions that make safety guarantees valid.

We begin with the concept that makes all of this precise: ascribing value to information.