Interaction Data and Passive Observation › Coverage, Behavior Policy, and Interventions

Dataset size alone does not say whether the state-action pairs needed at deployment are represented. Two equal-sized datasets can therefore support very different target policies. Coverage formalizes this target-relative distinction.

Coverage connects the behavior policy and exploration strategy to world-model generalization.

For a chosen time index, or for an explicitly defined time-averaged occupancy, the data distribution factorizes as

where:

• $d^{\beta}(s,a)$: the state-action occupancy distribution under the behavior policy $\beta$
• $d^{\beta}(s,a,s')$: the joint distribution of state, action, and next state under the behavior policy $\beta$
• $d^{\beta}(s)$: the state-occupancy distribution induced jointly by the initial-state distribution $\rho(s_0)$, the behavior policy $\beta$, and repeated transitions through $P$
• $\beta(a \mid s)$: the conditional probability or density of selecting action $a$ in state $s$
• $P(s' \mid s,a)$: the environment transition kernel
• $\rho(s_0)$: the initial-state distribution over starting states

The initial-state distribution is therefore not generally the same object as $d^{\beta}(s)$. It influences later state occupancy through the behavior policy and the environment dynamics. With the environment fixed, data collection can change $d^{\beta}(s,a)$ through both $\rho$ and $\beta$. Without additional structural assumptions, transition behavior remains unidentified wherever the resulting occupancy assigns no probability.

The factorization has an important implication: coverage can be improved in two different ways, and they target different gaps. Changing $\rho$ moves where the agent starts, which can flood previously unvisited regions with data even if the behavior policy is unchanged. Changing $\beta$ moves where the agent goes once it starts, which can fill in the actions and transitions reachable from any start. A careful data-collection design usually combines both, because starting broadly without a policy that spans the action space still leaves action-side coverage gaps.

Concretely: if the behavior policy is "always push with 1 N of force," no amount of that data identifies how the object behaves at 10 N without structural assumptions. If the behavior policy is "grasp the same red mug," predictions for other mugs depend on generalization beyond the observed object distribution. Causal confusion in imitation learning is related but distinct: a learner can select a predictive correlate of the expert's action rather than its cause, and the resulting policy fails after its own actions shift the observation distribution. Targeted interventions can distinguish the competing causal explanations; merely increasing sample count under the same correlations may not.

Formally, an offline model-based RL setting has a fixed dataset $\mathcal{D}$ sampled from $d^{\beta}$ and cannot collect more during offline training. The world model is trained on $\mathcal{D}$, then used inside a planner that evaluates candidate target policies. The target policy's state-action distribution $d^{\pi}$ is not generally the training distribution. Coverage is therefore one necessary part of assessing accuracy at planner queries, alongside sample size, model specification, and optimization error.

This leads to a clean diagnostic:

• In-support planning. The planner queries $(s, a)$ inside $\operatorname{supp}(d^{\beta})$, the support of the behavior-policy state-action distribution. This avoids pure support extrapolation, although finite data, partial observability, and model misspecification can still make predictions inaccurate.
• Near-support planning. The planner queries $(s, a)$ just outside $\operatorname{supp}(d^{\beta})$. Error may grow with distance from the data, but smooth growth is an inductive assumption rather than a guarantee. Any error can compound over a rollout horizon.
• Far-support planning. The planner queries $(s, a)$ far outside $\operatorname{supp}(d^{\beta})$. The data do not constrain those predictions. They may be wrong yet confident, giving the planner an opportunity to exploit model error.

The pattern is so common that modern offline MBRL often penalizes the planner for querying off-support actions, or ensembles several models and treats disagreement as a coverage signal. We'll pick this up in Part VIII: Decision-Centric Research Lineages, where ensembles and pessimistic planning are central topics. Pessimism, in this reading, is a way of encoding coverage as a preference: when the model is uncertain, prefer the plan that stays closer to the data.

Coverage can create a chicken-and-egg problem: the planner wants to visit states the dataset never saw, but the behavior policy did not collect them. One response is intervention: actively choose actions that populate the missing region. Here intervention is a data-collection strategy, using controlled action choices to distinguish outcomes that observational correlations leave ambiguous.

Three canonical strategies, revisited in Part VII, Ch 5: Active Perception, Dual Control, and Exploration, are:

• Random or entropy-maximizing exploration. For a bounded action space and a specified base measure, a uniform policy maximizes one-step action entropy. It does not guarantee uniform state coverage because the dynamics determine which states those actions reach.
• Uncertainty-driven exploration. Use a current model's uncertainty proxy, such as ensemble disagreement or estimated information gain, to choose actions. The strategy helps only when the proxy tracks epistemic error; correlated models can agree and still be wrong.
• Goal- or task-conditioned exploration. Deliberately probe regions relevant to a known task family. This concentrates coverage where it is expected to matter and leaves other regions sparse, which is acceptable only if the deployment family remains within scope.

Each strategy tries to improve coverage within a region of interest but can leave other regions sparse. Coverage is always relative to a target distribution. A random explorer may leave the task-relevant corner barely populated, while a task-conditioned explorer may miss rare events. Which failure matters more depends on the downstream queries.