Active Perception, Dual Control, and Exploration › Worked Example: A Gridworld with an Informative Sensor
World Models HandbookAll chapters Active Perception, Dual Control, and Exploration › Worked Example: A Gridworld with an Informative SensorOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We will construct a small partially observable gridworld in which the agent must reach one of several targets while its identity is uncertain, and it can optionally take an inspect action that reveals a noisy label. Then we will compare three policies: greedy reward, a thresholded curiosity heuristic, and a dual controller that trades task value against an optional information bonus. The worked example is deliberately tiny so we can compute the exact belief and information gain and see the tradeoff numerically.
The environment
The world is a grid. There are two candidate targets, A and B, placed at fixed cells. The agent knows the layout but not which target is rewarded; its prior is . It can move or inspect. An inspection costs reward units and reports the true binary label with probability and the other label with probability . Moving onto a target yields if it is rewarded and otherwise; movement itself has zero cost. The episode ends at either target. The implementation uses fixed paired episode conditions so policies face the same target identities and observation noise.
The next cells implement the environment, the exact Bayesian update, and the information gain.
## Gridworld geometry
GRID = 5
TARGETS = {"A": (0, 4), "B": (4, 4)} # (row, col) corners
START = (4, 0)
ACTIONS = ["up", "down", "left", "right", "inspect"]
DELTAS = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}
def step_cell(cell, action):
"""Return the new cell after a move; walls clip the move."""
r, c = cell
dr, dc = DELTAS[action]
nr, nc = (
int(np.clip(r + dr, 0, GRID - 1)),
int(np.clip(c + dc, 0, GRID - 1)),
)
return (nr, nc)
## Prior belief over which target is rewarded
PRIOR = np.array([0.7, 0.3]) # index 0 -> target A, index 1 -> target B
INSPECT_ACC = 0.8
INSPECT_COST = 0.18def bayes_update(belief, obs_index):
"""Posterior over target identity after observing obs_index in {0,1}."""
lik = np.where(np.arange(2) == obs_index, INSPECT_ACC, 1.0 - INSPECT_ACC)
posterior = belief * lik
return posterior / posterior.sum()
def inspect_observation_probs(belief):
"""Predictive P(o | belief) for the two possible observations."""
acc = INSPECT_ACC
p_o0 = belief[0] * acc + belief[1] * (1 - acc)
p_o1 = belief[0] * (1 - acc) + belief[1] * acc
return np.array([p_o0, p_o1])The step_cell function encodes the deterministic movement dynamics. The code clips at walls so the world is fully specified without boundary errors, and the target-identity belief update is exact because the hidden variable has two enumerated hypotheses and the observation model is known.
bayes_update is the belief-Bayes step; it multiplies the prior by the observation likelihood and renormalizes, which is the exact Bayesian posterior for a single observation. inspect_observation_probs is the denominator that the belief-MDP transition needs, computed by summing over the two possible target identities weighted by the prior. With these two pieces in place we can compute the exact information gain of inspecting.
def entropy(belief):
b = belief[belief > 0]
return float(-(b * np.log(b)).sum())
def information_gain_inspect(belief):
"""Exact expected information gain (mutual information) of inspecting."""
p_o = inspect_observation_probs(belief)
expected_posterior_entropy = 0.0
for o in range(2):
expected_posterior_entropy += p_o[o] * entropy(bayes_update(belief, o))
return entropy(belief) - expected_posterior_entropyThis is the entropy-reduction definition, computed exactly: the function sums the posterior entropy over the two possible observations, weighted by their predictive probabilities, and subtracts that average from the prior entropy. The next cells evaluate it on a few beliefs and confirm the two sanity checks: nonnegativity, and the collapse to zero when the belief is already a point mass.
test_beliefs = [
np.array([0.5, 0.5]),
np.array([0.7, 0.3]),
np.array([0.9, 0.1]),
np.array([1.0, 0.0]),
]
entropies = [entropy(b) for b in test_beliefs]
info_gains = [information_gain_inspect(b) for b in test_beliefs] belief | entropy | info gain (inspect)
[0.5 0.5] | 0.6931 | 0.1927
[0.7 0.3] | 0.6109 | 0.1637
[0.9 0.1] | 0.3251 | 0.0727
[1. 0.] | -0.0000 | -0.0000For this fixed symmetric binary sensor, the uniform belief has the highest raw information gain, while a point-mass belief has zero. This is a property of the channel and the information objective; it does not imply that inspection has the highest task value at maximum entropy in every problem.
The same relationship appears as a continuous curve when we sweep the prior probability over the full simplex, which shows how the information gain from inspection degrades as the agent's belief converges.
Both curves peak at the uniform belief and fall to zero at the point masses for this symmetric channel, which is the continuous form of the sanity check we just ran.
Now we can write the three policies. Each policy returns an action given the agent cell and the current belief. greedy moves toward the currently most probable target; curious inspects whenever the information gain exceeds a threshold; dual compares immediate commitment with a one-probe-then-commit score that adds an optional information bonus.
def move_toward(cell, target):
r, c = cell
tr, tc = target
if r != tr:
return "up" if tr < r else "down"
if c != tc:
return "left" if tc < c else "right"
return "inspect" # already on target; return an admissible fallback
## This cell defines policy functions only; it performs no training, simulation,
## or I/O. It is a presentation-only helper that other figure cells consume, so
## it is safe to rerun under any theme.
def greedy_policy(cell, belief):
target = TARGETS["A"] if belief[0] >= belief[1] else TARGETS["B"]
return move_toward(cell, target)
def curious_policy(cell, belief, threshold=1e-3):
if cell == START and information_gain_inspect(belief) > threshold:
return "inspect"
return greedy_policy(cell, belief)
def commit_value(belief):
"""Expected terminal reward from choosing the more likely target now."""
return 2.0 * float(np.max(belief)) - 1.0
def probe_then_commit_value(belief):
"""Expected terminal reward after one inspect, less its reward cost."""
p_o = inspect_observation_probs(belief)
value = sum(
p_o[o] * commit_value(bayes_update(belief, o)) for o in range(2)
)
return value - INSPECT_COST
def dual_policy(cell, belief, beta=0.25):
"""Inspect when decision value plus a scaled information bonus justifies it."""
if cell == START:
inspect_score = probe_then_commit_value(
belief
) + beta * information_gain_inspect(belief)
if inspect_score > commit_value(belief):
return "inspect"
return greedy_policy(cell, belief)The decision-aware part compares committing now with inspecting once and then committing, including inspection cost. The optional term deliberately distorts that task value toward raw information. At , this is a one-probe value-of-information calculation. Positive lets us see when a generic information bonus causes over-inspection. The policy re-evaluates after every observation and moves once another probe no longer clears its combined task-value-plus-information score; at , this reduces to the cost-aware one-probe test.
We now roll out all three policies on the same environment and track the result.
def rollout(
policy, true_target, observation_uniforms, max_steps=30, belief=None
):
cell = START
belief = PRIOR.copy() if belief is None else belief.copy()
total_reward, log = 0.0, []
for t in range(max_steps):
action = policy(cell, belief)
if action == "inspect":
total_reward -= INSPECT_COST
o = (
true_target
if observation_uniforms[t] < INSPECT_ACC
else 1 - true_target
)
belief = bayes_update(belief, o)
log.append(("inspect", entropy(belief)))
else:
cell = step_cell(cell, action)
name = ["A", "B"][true_target]
if cell == TARGETS[name]:
total_reward += 1.0
log.append(("hit", 0.0))
break
other = ["A", "B"][1 - true_target]
if cell == TARGETS[other]:
total_reward -= 1.0
log.append(("miss", 0.0))
break
return total_reward, log
N_EPISODES = 1000
experiment_rng = np.random.default_rng(7)
episode_targets = (experiment_rng.random(N_EPISODES) >= PRIOR[0]).astype(int)
episode_observations = experiment_rng.random((N_EPISODES, 30))
results = {}
for name, pol in [
("greedy", greedy_policy),
("curious", curious_policy),
("dual", dual_policy),
]:
rewards = [
rollout(pol, target, uniforms)[0]
for target, uniforms in zip(episode_targets, episode_observations)
]
results[name] = np.mean(rewards)
## Every policy uses the same target labels and observation uniforms.The targets are sampled from the stated prior once, and every policy receives the same target labels and potential observation noise. Greedy commits immediately. Curious keeps inspecting while raw information gain exceeds its threshold. The dual policy compares task value before adding its optional information bonus.
mean reward over 1000 paired episodes, greedy: +0.406 mean reward over 1000 paired episodes, curious: -0.635 mean reward over 1000 paired episodes, dual: +0.481
The comparison separates three objectives. Greedy avoids sensing cost but sometimes chooses the wrong target. The thresholded curiosity heuristic gathers far more evidence than the terminal decision needs and pays for it. The decision-aware policy uses inspection only when its expected improvement exceeds its cost. These are seeded illustrative results, not a benchmark.
The tradeoff becomes explicit when we sweep and the prior confidence, plotting both in the same two-column figure so the interaction is visible. Here the value-of-information story from the theory section emerges as numbers.
The first sweep varies while holding the paired episodes fixed. The second varies a calibrated prior and compares immediate commitment with the decision-aware one-probe rule at . Each confidence level gets its own deterministic seed.
confidences = [0.5, 0.6, 0.7, 0.8, 0.9, 0.95]
confidence_results = [evaluate_confidence(c) for c in confidences]
greedy_by_conf = [row[0] for row in confidence_results]
dual_by_conf = [row[1] for row in confidence_results]
Reading the left panel: small bonus weights do not change the task-optimal one-probe decision. Large weights purchase additional information after it stops paying for itself, so extrinsic return falls. This is not evidence for a universal best ; the threshold depends on reward scale, sensing cost, prior, and sensor channel.
Reading the right panel: with a calibrated prior, immediate commitment improves as confidence rises. Decision-aware inspection helps while its expected terminal-reward improvement exceeds . Under the terminal rewards, this is equivalent to reducing target-selection error probability by more than . The optimal amount of sensing is a property of the world, belief, decision, reward scale, and sensing cost together.
The two panels share a hidden variable: the gap between raw information gain and decision-relevant value. At each prior we compute both quantities from the same sensor, but we keep their units separate.
def gross_value_of_information(belief):
"""Expected terminal-reward gain from a free inspect before target choice."""
p_o = inspect_observation_probs(belief)
conditioned = sum(
p_o[o] * commit_value(bayes_update(belief, o)) for o in range(2)
)
return conditioned - commit_value(belief)
## This showcase curve cell uses only pure computation over the two-state
## simplex, so it is safe to rerun under any theme.
voi_curve = []
for p in prior_grid:
b = np.array([p, 1.0 - p])
voi_curve.append(gross_value_of_information(b))
voi_curve = np.array(voi_curve)
The panels deliberately do not share a numerical axis. The left describes the observation channel; the right describes the target-choice utility. Inspection has positive net value only where the reward-valued curve lies above the cost line. Changing terminal rewards would rescale the right panel without changing the left.
We now add a known forbidden cell and test a one-step action filter. This is intentionally narrower than a synthesized formal shield: it certifies only that the returned action does not enter the known hazard on the next transition.
HAZARD = (2, 2)
def is_hazard(cell):
return cell == HAZARD
def safe_actions(cell):
"""Return actions whose one-step successor is not the known hazard."""
moves = [a for a in DELTAS if not is_hazard(step_cell(cell, a))]
return moves + ["inspect"]
def shielded_move(cell, candidate_action, target=TARGETS["A"]):
"""One-step filter for this known deterministic grid."""
if candidate_action == "inspect":
return candidate_action
if candidate_action in safe_actions(cell):
return candidate_action
safe_moves = [a for a in safe_actions(cell) if a != "inspect"]
if not safe_moves:
return "inspect"
return min(
safe_moves,
key=lambda a: (
abs(step_cell(cell, a)[0] - target[0])
+ abs(step_cell(cell, a)[1] - target[1])
),
)
## Four adversarial proposals approach the hazard from each neighboring cell.
filter_cases = [
((3, 2), "up"),
((2, 1), "right"),
((1, 2), "down"),
((2, 3), "left"),
]
unfiltered_hits = sum(
is_hazard(step_cell(cell, action)) for cell, action in filter_cases
)
filtered_actions = [
shielded_move(cell, action) for cell, action in filter_cases
]
filtered_hits = sum(
is_hazard(step_cell(cell, action))
for (cell, _), action in zip(filter_cases, filtered_actions)
if action != "inspect"
)
assert unfiltered_hits == len(filter_cases)
assert filtered_hits == 0
assert all(
action in safe_actions(cell)
for (cell, _), action in zip(filter_cases, filtered_actions)
)The filter constructs the admissible one-step action set first, then either accepts the proposal or replaces it with a verified safe fallback rather than projecting it in action space. The assertions exercise an easy-to-miss failure mode: a fallback must itself be checked.
hazard-entering proposals without filter: 4 hazard-entering actions after filter: 0 verified fallback actions: ['right', 'up', 'up', 'up']
All four unfiltered proposals enter the hazard; all four filtered actions avoid it. This establishes only a one-step invariant for the deterministic grid and known hazard. It does not establish route completion, robustness to unknown hazards, or safety under model mismatch. A formal shield needs a sound transition abstraction or model, a specification, and a proof that a safe action remains available over the relevant horizon.