Offline and Conservative Model-Based RL

Picture a warehouse robot that has been running for three years. It has driven several thousand kilometres, followed human-designed routes, bumped into the same four pallets, and logged every state, action, and outcome. You now have a large dataset. You have no way to collect more. Can those logs support a better controller than the one that produced them?

The answer is not obvious and not always yes. The dataset was produced by some behaviour policy, perhaps a careful human operator or an older controller. Every transition reflects choices that policy made. If it never tried to squeeze between two pallets, the dataset contains no direct evidence about what happens when you try. If it never braked hard on a wet floor, the dataset is silent on that too. Improving the controller may require different actions, and those actions may lack coverage in the logs. That is the tension we spend the chapter unpicking.

Every model-based method we have covered so far in Part VIII: Decision-Centric Research Lineages assumed a closed loop. PETS, MBPO, Dreamer, MuZero, and TD-MPC all interleave acting in the world with learning from the consequences. The world model is wrong in a hundred small ways, the agent tries something, reality corrects it, and the model improves. That correction signal is the engine.

When the agent acts, the observed successor provides a targeted new sample at the state-action pair it tried. In a stochastic environment, one outcome is not a measurement of how wrong the model's predictive distribution is; repeated observations help distinguish environmental randomness from systematic model error. Over many interactions a well-designed learning loop can concentrate model corrections where the agent visits. The state distribution induced by the policy and the region where the model improves can then move together. Offline learning loses that feedback loop.

Offline reinforcement learning removes that engine during training. The agent gets one fixed dataset and no new interaction while learning its policy. Deployment may provide feedback later, or an offline-to-online phase may collect more data, but neither can correct an unsupported decision before this offline training stage ends. The policy must therefore be selected using evidence already in the log.

That single change breaks almost everything. Two failure modes follow directly from it, and they are worth stating plainly before any mathematics:

• The log alone does not constrain the model off its support. If your dataset never contains the state-action pair $(s, a)$, competing predictions there can fit the logged transitions equally well. A model that says "you teleport to the goal" and a model that says "you fall into a pit" may both fit the training set. Architecture, prior knowledge, or justified structural assumptions can distinguish them, but the log provides no direct target at that pair to pull the model toward the truth.
• The optimiser can exploit unsupported optimistic estimates. A policy improvement step that maximises estimated value favours regions with high estimates. If the model happens to be optimistic off-support, that step can select a policy that visits a region where the log provides little evidence. This is model exploitation, and it is not a defect of one algorithm: it is a risk whenever policy selection relies on estimates that are insufficiently grounded.

The mitigations address those two risks. We can quantify how uncertain the model is (ensembles, counts, densities). We can be pessimistic, subtracting an uncertainty penalty to make poorly supported choices less attractive. Or we can constrain the policy itself to stay near the behaviour that produced the data. The rest of the chapter develops all three ideas, illustrates uncertainty and pessimism on a gridworld, and then turns to the meta-problem that offline RL never escapes: how do you evaluate a policy you cannot run?

This section fixes notation, states a coverage assumption that can support offline learning, and derives the simulation lemma, which is the single most useful equation in the chapter. It explains why a model that looks accurate can produce a badly wrong value function, and it draws the line between model-based and model-free approaches to the offline problem. Later we will distinguish coverage from justified structural assumptions that can also constrain unobserved outcomes.

We have a Markov decision process $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma, d_0)$ as introduced in Part II: Inference, Decisions, and Control. To keep the comparison with learned models clean, we will assume throughout that the state is fully observed and the MDP is stationary. Neither assumption is free. Under partial observability the effective state is a belief, and the coverage problem becomes harder because two observation histories can be indistinguishable while requiring different actions. We will flag where that matters.

The stationarity assumption matters too. If the environment drifts over time, last year's transitions may not describe the world the deployed policy will face. In the warehouse example, a reorganised floor plan changes the dynamics represented by the logs. Here we assume no such drift: training and deployment share the same environment.

Instead of interacting, we are handed a dataset

$$\mathcal{D} = \{(s_i, a_i, r_i, s_i')\}_{i=1}^{N}, \qquad (s_i, a_i) \sim \rho_\beta, \quad \mathbb{E}[r_i\mid s_i,a_i]=r(s_i,a_i), \quad s_i' \sim P(\cdot \mid s_i, a_i),$$

where:

• $\mathcal{D}$: the fixed dataset of $N$ transitions
• $s_i$: the $i$-th observed state
• $a_i$: the $i$-th observed action taken in that state
• $r_i$: the observed reward, whose conditional mean is $r(s_i,a_i)$
• $s_i'$: the next state, drawn from the transition distribution $P(\cdot\mid s_i, a_i)$
• $\rho_\beta$: the behaviour distribution, the state-action distribution induced by whatever process generated the logs

The behaviour distribution might come from a human demonstrator, a hand-tuned controller, a previously deployed policy, or five such policies mixed together. We do not need to know which. We only need the samples. Notice that the dataset records actions and their outcomes but never explains them. Nothing tells us why the behaviour policy chose a given action, which matters later when we want to imitate it in states where the model is uncertain.

The objective is unchanged from online RL. We want a policy $\pi$ maximising

$$J(\pi) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \;\middle|\; s_0 \sim d_0,\; a_t \sim \pi(\cdot\mid s_t),\; s_{t+1}\sim P(\cdot\mid s_t, a_t)\right].$$

where:

• $J(\pi)$: the expected discounted return of policy $\pi$
• $\gamma$: the discount factor in $[0,1)$
• $r(s_t, a_t)$: the reward received at step $t$
• $d_0$: the initial-state distribution, with $s_0\sim d_0$
• $\pi(\cdot\mid s_t)$: the policy, giving the action distribution at state $s_t$
• $P(\cdot\mid s_t, a_t)$: the environment transition distribution

The difference from online RL is entirely in what we are allowed to do while searching for $\pi$. Online, we can test $\pi$, observe the return, and adjust. Offline, that loop is severed: the objective is still defined against the true environment, but we can only ever evaluate candidate policies against our beliefs about it.

It is convenient to work with discounted state-action occupancy measures rather than raw trajectories. For a policy $\pi$ with initial distribution $d_0$, define the mass of any measurable state-action set $B$ by

$$d^\pi(B) = (1-\gamma)\sum_{t=0}^{\infty} \gamma^{t}\, \Pr((s_t,a_t)\in B \mid \pi, d_0).$$

where:

• $d^\pi(B)$: the discounted occupancy mass that policy $\pi$ assigns to set $B$
• $\Pr((s_t,a_t)\in B\mid\pi, d_0)$: the probability of visiting $B$ at step $t$ under policy $\pi$ starting from $d_0$
• $1-\gamma$: the normalising factor that makes the sum a probability distribution

The $(1-\gamma)$ factor makes $d^\pi$ a probability distribution over $\mathcal{S}\times\mathcal{A}$. In a discrete space, $d^\pi(s,a)$ abbreviates the mass of the singleton $\{(s,a)\}$; in a continuous space, it denotes a density only when one exists with respect to a stated reference measure. Why bother with this object rather than working with the trajectory distribution directly? Because an occupancy measure summarises "where the policy spends its time," weighting near-term visits more heavily than far-future ones. It converts the trajectory of the policy into a static picture of the space, which is exactly what we need to compare two policies and to reason about how much their visited regions overlap. With this normalisation, the objective is a simple expectation:

$$J(\pi) = \frac{1}{1-\gamma}\,\mathbb{E}_{(s,a)\sim d^\pi}\big[r(s,a)\big].$$

Now compare $d^\pi$ with $\rho_\beta$. If a policy spends time where the behavior distribution has substantial mass, the data can inform those choices; where it has none, learning requires additional structure. For the displayed ratio, read both quantities as masses in a discrete space or densities with respect to a common reference measure in a continuous space. This intuition is captured by the concentrability coefficient:

$$C(\pi) \;=\; \sup_{s,a} \; \frac{d^\pi(s,a)}{\rho_\beta(s,a)} .$$

where:

• $C(\pi)$: the concentrability coefficient of policy $\pi$
• $d^\pi(s,a)$: the discounted occupancy of $(s,a)$ under policy $\pi$
• $\rho_\beta(s,a)$: the behaviour distribution's density at $(s,a)$
• $\sup_{s,a}$: the supremum over state-action pairs in the discrete case, or the essential supremum of the density ratio in the continuous case

Define the ratio as infinite when the target occupancy assigns positive mass to a region where the behavior distribution assigns zero; zero-over-zero pairs do not affect it. In continuous spaces, this requires absolute continuity and an essentially bounded density ratio. Thus finite $C(\pi)$ requires the occupancy of that policy to be supported by the behavior distribution. Some offline-RL guarantees assume coverage of an optimal policy, $C^\star=C(\pi^\star)<\infty$, together with further statistical and algorithmic conditions. Covering one optimal policy is weaker than covering every policy, but it does not by itself make any particular learning algorithm successful. A small coefficient says the behavior distribution gives reasonable weight to what the policy visits; a large one signals weak coverage.

Coverage is one way to make offline learning identifiable. Without coverage or justified structure linking unobserved outcomes to observed ones, no method can guarantee good decisions uniformly over the unrestricted MDP class.

The coefficient compares state-action occupancy densities. The next picture simplifies them to their state marginals: if the target visits states the behaviour never reaches, then it necessarily visits unsupported state-action pairs and $C(\pi)$ is infinite. A gap confined to actions at otherwise covered states would also matter, but this state-only picture does not show it.

Now suppose we learn a model $\hat M = (\hat P, \hat r)$ from $\mathcal{D}$ and use it in place of the environment. How wrong is the resulting value function? The answer is the simulation lemma, and it deserves a careful derivation because it is the theoretical heart of the whole chapter.

Fix a deterministic policy $\pi$ for this derivation; a stochastic policy gives the same identity after averaging each action-dependent term over $a\sim\pi(\cdot\mid s)$. Write $V^\pi_M$ for its value under the true MDP and $V^\pi_{\hat M}$ for its value under the model. Define the gap $\Delta(s) = V^\pi_M(s) - V^\pi_{\hat M}(s)$. Expanding both Bellman equations,

$$\begin{aligned} V^\pi_M(s) &= r(s,\pi(s)) + \gamma\, \mathbb{E}_{s'\sim P(\cdot\mid s,\pi(s))}\big[V^\pi_M(s')\big],\\ V^\pi_{\hat M}(s) &= \hat r(s,\pi(s)) + \gamma\, \mathbb{E}_{s'\sim \hat P(\cdot\mid s,\pi(s))}\big[V^\pi_{\hat M}(s')\big]. \end{aligned}$$

Subtracting the second from the first and adding and subtracting a term that evaluates $V^\pi_M$ under the model,

$$\Delta(s) = \underbrace{\big(r - \hat r\big)}_{\text{reward error}} + \gamma \underbrace{\Big(\mathbb{E}_{P} V^\pi_M - \mathbb{E}_{\hat P}\, V^\pi_M\Big)}_{\text{transition error, evaluated on the true value}} + \gamma\, \mathbb{E}_{\hat P}\big[\Delta(s')\big].$$

where:

• $\Delta(s) = V^\pi_M(s) - V^\pi_{\hat M}(s)$: the value gap at state $s$
• $r - \hat r$: the reward-model error, abbreviated $r(s,\pi(s)) - \hat r(s,\pi(s))$
• $\mathbb{E}_P V^\pi_M - \mathbb{E}_{\hat P}\, V^\pi_M$: the transition error, evaluated using the true value function $V^\pi_M$
• $\mathbb{E}_{\hat P}[\Delta(s')]$: the expected continuation of the gap under the model's dynamics

The adding-and-subtracting trick is the algebraic seed of the whole result. We insert $\mathbb{E}_P V^\pi_M - \mathbb{E}_{\hat P} V^\pi_M$ not because it was there already, but to isolate the reward mismatch and the transition mismatch as a single "local inconsistency" while leaving a clean recursion for the remaining gap. Everything that follows is a matter of chasing where that local inconsistency gets weighted.

The first two terms do not depend on $\Delta$; call their sum the local inconsistency $\ell(s)$. The recursion $\Delta = \ell + \gamma \mathbb{E}_{\hat P}\Delta$ unrolls along the trajectory distribution generated by the model. Write $d^\pi_{\hat M}(s,a)$ for the model's normalised discounted state-action occupancy and $d^\pi_{\hat M}(s)$ for its state marginal, both induced by the learned model $\hat M$ rather than the true MDP. Averaging the initial-state gap over $s_0\sim d_0$ gives

$$\boxed{\;J_M(\pi)-J_{\hat M}(\pi) \;=\; \frac{1}{1-\gamma}\,\mathbb{E}_{s\sim d^\pi_{\hat M}}\big[r(s,\pi(s)) - \hat r(s,\pi(s)) + \gamma\delta(s)\big]\;}$$

Here $J_M(\pi)=\mathbb E_{s_0\sim d_0}V_M^\pi(s_0)$, $J_{\hat M}(\pi)=\mathbb E_{s_0\sim d_0}V_{\hat M}^\pi(s_0)$, and $\delta(s)=\mathbb{E}_{P(\cdot\mid s,\pi(s))}[V^\pi_M(s')] - \mathbb{E}_{\hat P(\cdot\mid s,\pi(s))}[V^\pi_M(s')]$. A pointwise value-gap identity would instead use a model occupancy conditioned on that particular starting state; the displayed occupancy is averaged over $d_0$.

The boxed equation averages local errors over states visited by a fixed policy in the learned model, not necessarily in reality. If the model's dynamics place substantial occupancy in a region where its predictions are wrong, that region contributes heavily to the value gap. The identity does not say that error itself attracts a fixed policy toward that region.

Error compounding and model exploitation are distinct steps. The fixed-policy identity describes how one-step errors propagate through model rollouts. When a policy is then selected by maximizing learned-model value, it may favor optimistic, poorly supported regions; that selection can change the model occupancy in the identity. The two mechanisms can reinforce each other, but neither the identity nor a fixed-policy bound alone proves that the errors concentrate exactly where they are largest.

Bound the two error terms to get a usable inequality. Suppose the reward model is off by at most $\epsilon_r$ and the transition model has total-variation error at most $\epsilon_P$:

$$\big|r(s,a) - \hat r(s,a)\big| \le \epsilon_r, \qquad \big\|P(\cdot\mid s,a) - \hat P(\cdot\mid s,a)\big\|_{TV} \le \epsilon_P .$$

where $\epsilon_r$ bounds the reward error, $\epsilon_P$ bounds the total-variation transition error, and $R_{\max}$ is the maximum absolute reward. Since $|V^\pi_M(s')| \le R_{\max}/(1-\gamma)$ and $|\mathbb{E}_p f - \mathbb{E}_q f| \le 2\|p-q\|_{TV}\max|f|$,

$$\big|V^\pi_M(s) - V^\pi_{\hat M}(s)\big| \;\le\; \frac{\epsilon_r}{1-\gamma} \;+\; \frac{2\gamma R_{\max}\,\epsilon_P}{(1-\gamma)^2}.$$

where:

• $\epsilon_r$: the maximum reward-model error, $\epsilon_r \ge |r(s,a) - \hat r(s,a)|$
• $\epsilon_P$: the maximum total-variation transition-model error
• $R_{\max}$: the maximum absolute per-step reward
• $\gamma$: the discount factor

The $1/(1-\gamma)$ and $1/(1-\gamma)^2$ prefactors come from two separate sources, and it is instructive to see both. The reward error is a per-step bias paid every step, so summing it under discounting costs the horizon factor $1/(1-\gamma)$. The transition error is more expensive: total-variation error is compared against value functions bounded by $R_{\max}/(1-\gamma)$, and each step of the horizon multiplies the accumulated value bound by another factor, so the sum produces $1/(1-\gamma)^2$. The quadratic blow-up in the transition term is exactly the compounding effect, and it is the term that decides whether the bound is usable.

Now plug in an illustrative set of numbers. Take $R_{\max}=1$ and $\gamma = 0.99$. If rewards lie in $[0,1]$, then returns lie in $[0,100]$; under the weaker $|r|\leq 1$ assumption, returns lie in $[-100,100]$. Let the transition model have a uniform total-variation error upper bound of $\epsilon_P = 0.05$ per state-action pair. This is a distance between distributions, not a claim that 95% of predicted transitions are correct. The transition-error term gives

$$\frac{2 \times 0.99 \times 1 \times 0.05}{(1-0.99)^2} \;=\; \frac{0.099}{10^{-4}} \;=\; 990,$$

With $\epsilon_r=0.01$, the reward-error term adds 1, so the full bound is 991. Even the transition term alone exceeds a range-based upper bound of 201 on the value difference when true rewards satisfy $|r|\leq 1$ and the learned reward may differ by up to $\epsilon_r=0.01$. The bound is therefore vacuous in this example: a uniform 0.05 per-step TV bound does not yield a useful guarantee at this discount factor.

Plotting the two error terms against the discount factor shows which one drives the blow-up.

The moral is not that the bound is bad. The bound is correct, and it is telling you something true: worst-case, there is no useful guarantee. A useful guarantee requires additional information, such as coverage of the policies under consideration or justified structure that bounds model error where they go. Pessimistic penalties and policy constraints can use such assumptions to limit unsupported choices; they do not themselves create coverage or certify a heuristic uncertainty score. This is why offline model-based RL is not simply "model-based RL without exploration."

It is worth emphasising that the numbers here are chosen to make the point sharp, not because $\epsilon_P = 0.05$ is a realistic worst case. A uniform TV upper bound of 0.05 permits that much transition error at any state-action pair, including those the policy visits; it does not say actual error is 0.05 everywhere. The vacuity comes from pairing that global error allowance with a long horizon. Shorten the horizon ($\gamma \to 0$) and the bound tightens quickly; make the model extremely accurate only where the policy goes and the uniform bound overstates the damage. Both of these escapes correspond to real assumptions we can make, and both are what the algorithms in the next sections exploit.

Why bother with a model at all, given the risk? Because of the amplification factor. With $N$ real transitions you can train a value function on $N$ samples, or you can train a model on $N$ samples and generate many imagined transitions from it. If the model is accurate on the region the policy cares about, those rollouts can help. MBPO made this argument in the online setting, as discussed in Part VIII: Decision-Centric Research Lineages. Online interaction can expose a wrong prediction and provide data for correction, though it does not guarantee safe exploration or eventual recovery. Offline, the fixed log cannot supply that targeted correction; repeated synthetic rollouts retain the model's unsupported assumptions.

The natural worry is a mismatch between where the model is accurate and where an improved policy might go. The dataset reflects $\rho_\beta$, while an unconstrained improvement step can favor actions with little coverage. Improvement need not leave support: a mixed or suboptimal behaviour dataset may already contain better actions. The methods below try to use that opportunity without letting the optimiser exploit unsupported estimates.

This section defines what "out of distribution" means for a learned model and why the action dimension is usually where the trouble starts. It builds the machinery for quantifying epistemic uncertainty, then constructs a gridworld where the failure is visible and unambiguous.

A dataset of continuous-control trajectories often covers a thin tube in joint state-action space. Even millions of transitions may leave reachable $(s,a)$ pairs poorly sampled. A robot moving through a hallway traces a narrow ribbon inside a higher-dimensional joint space. Three useful views of extrapolation arise; they overlap rather than partition the space.

• State OOD. The query state lies outside, or in a poorly covered part of, the behaviour state's distribution. This can happen when the policy drives into a new configuration; exact equality with a logged continuous state is not the test.
• Action OOD. The state is familiar but the proposed action is poorly supported conditional on that state. This is subtler: driving logs may cover a speed well but contain little evidence about full braking at that speed.
• Joint OOD. The pair $(s,a)$ is missing or rare in the logged joint distribution, even when the state and action have each appeared separately. State OOD or action OOD also makes the pair OOD; the distinctive case is a novel pairing of familiar marginals.

Action OOD is the one people underestimate. Consider a behaviour policy that is nearly deterministic, as some demonstration datasets are. In a well-covered state, the marginal $\rho_\beta(s)$ may be large, but conditional on $s$ the action distribution is a narrow spike. The model has evidence for only a narrow range of actions at that state, even if it has many transitions. If an improvement step proposes an action 2 standard deviations away from the spike, the transition model may have little evidence about the consequence, but the value function still returns a number, and the optimiser may still trust it.

These views interact. In a low-dimensional example, a state far from logged states may be visibly unusual; in high dimensions, that judgment is harder. A state-only density model cannot detect a poorly supported action at a familiar state. A familiar-looking state and a familiar-looking action can also form a rare pair, so checking only $p(s)$ cannot establish joint state-action coverage. The relevant question for a transition model is whether the pair has enough nearby evidence.

In reconstructive latent world models from Part III: Representing Agents and Worlds, an encoder maps observations into a latent space and a decoder reconstructs observations. Other architectures, including MuZero, need not have a decoder. If planning drives a latent state far beyond training support, its predicted transitions and values can become unreliable even though the vectors remain numerically valid. A model may not raise an exception or produce a NaN when this happens, so checking only numerical validity will miss the coverage problem.

We will compare five approaches to estimating or bounding uncertainty. They differ in cost, calibration, and how badly they can fail.

A schematic decomposition makes the distinction concrete before the individual estimator families are described.

No single family dominates: the choice depends on whether you need speed, a defensible statistical statement, or robustness to shared model bias.

Ensembles and disagreement. Train $K$ dynamics models $\{\hat P_{\theta_1}, \ldots, \hat P_{\theta_K}\}$ with different random initialisations and different bootstrapped data subsets. Each predicts a next-state mean $\mu_k(s,a)$. Then

$$u(s,a) = \max_k \Big\| \mu_{\theta_k}(s,a) - \bar\mu(s,a)\Big\|_2, \qquad \bar\mu(s,a) = \frac{1}{K}\sum_{k=1}^{K}\mu_{\theta_k}(s,a).$$

where:

• $u(s,a)$: the disagreement-based uncertainty score at $(s,a)$
• $\mu_{\theta_k}(s,a)$: the next-state mean predicted by ensemble member $k$ with parameters $\theta_k$
• $\bar\mu(s,a)$: the mean prediction averaged over the $K$ members
• $\|\cdot\|_2$: the Euclidean norm

This is one possible disagreement score, not MOPO's practical penalty. MOPO uses the maximum Frobenius norm of a member's predicted Gaussian scale matrix across its probabilistic model ensemble, which its authors describe as a maximum standard-deviation score. Ensemble disagreement can be useful, but it is not a calibrated posterior uncertainty; members with shared architecture and data may agree even when they are all wrong. Nor does predictive variance automatically isolate epistemic uncertainty from environmental randomness. The disagreement captures how much the members differ from each other, not how far the ensemble as a whole is from the truth. If all members are biased in the same way, the disagreement stays small while the error stays large.

Count-based uncertainty. In discrete or discretisable spaces, uncertainty is a function of the visit count $n(s,a)$. A common form is

$$u(s,a) = \frac{\alpha}{\alpha + n(s,a)}, \qquad \text{or} \qquad u(s,a) = \frac{1}{\sqrt{n(s,a)+1}}.$$

where:

• $u(s,a)$: the uncertainty score at $(s,a)$
• $n(s,a)$: the number of times $(s,a)$ was visited in the dataset
• $\alpha$: a smoothing prior (set to 1 in our implementation below)

The second has the $1/\sqrt n$ scaling of many concentration bounds. It is not, without a specified prior and observation-noise variance, an exact posterior standard deviation. Count-based scores are cheap and interpretable when counts are meaningful; we use one below to isolate the penalty mechanism from neural-network training noise. Their weakness is that "count" requires a definition of "the same event," and in continuous state spaces that definition is a discretisation, which introduces its own choices.

Density and likelihood models. Fit $\hat\rho_\beta(s,a)$ with a normalising flow or a VAE, then use a quantity such as $-\log \hat\rho_\beta(s,a)$ as a proxy for low data density. It is not automatically a calibrated bound on transition-model error or a reliable test of support. BEAR instead uses behavior-action samples and an MMD constraint in policy improvement. FisherBRC learns a behavior density for its critic decomposition $Q_\theta(s,a)=O_\theta(s,a)+\log\hat\mu(a\mid s)$ and regularizes action gradients of the offset; neither method defines this density score as a world-model reward penalty. Likelihood is not the same as support: a generative model can give high likelihood to out-of-distribution inputs, as likelihood-based OOD studies demonstrate.

Bayesian neural networks and variational inference. Place a prior over model parameters and use a posterior predictive distribution. These methods introduce inference and calibration choices that differ from those of a bootstrap ensemble; no single family wins across tasks and model classes. PILCO's Gaussian-process dynamics, discussed in Part VIII: Decision-Centric Research Lineages, provide posterior uncertainty under the GP model assumptions. Naive exact GP fitting has cubic time cost in the number of training points, so that particular implementation does not scale cheaply to large datasets.

Spectral and Lipschitz-based bounds. A smooth learned model alone does not guarantee a small error outside the data. If the error function is known to be Lipschitz, or both true and learned dynamics have compatible known Lipschitz bounds, an error measured near a query point can be extended with a distance term. Such a result is deterministic under its assumptions, though its constants may be too loose to guide planning. The needed quantity is a bound on how quickly error can change, not merely on how quickly the learned prediction changes.

Now we build something concrete. The goal is not to reproduce a benchmark. It is to construct the smallest system in which we can see the mechanism with our own eyes, compute everything exactly, and check the answer against ground truth. Minimality matters here. Offline RL failures on real benchmarks are opaque: you see a poor return, but you cannot easily separate model error from coverage. In a small grid we can isolate those two factors directly.

The environment is a $7 \times 60$ grid. Row 3 is a corridor that runs the full width; the goal sits at the far right end of that row. The grid edge acts as a wall. The goal is absorbing with zero reward, and reaching it pays 1.0. The discount factor is $\gamma = 0.99$.

The behaviour policy is a random walk along row 3, moving right with probability 0.7 and left with probability 0.3. It never leaves the corridor. This is the crucial design choice: the dataset covers one row out of seven, and covers it densely. Every other cell in the grid has never been seen.

The leftward moves matter. With a purely rightward policy, each of the 59 pre-goal corridor cells would be visited once per episode, while the absorbing goal would still receive 20 logged tail visits. The 30% leftward action instead makes the walk revisit earlier cells, producing a non-uniform pre-goal visit distribution.

The left edge clips a left action back to the same cell, while the rightward bias eventually carries most episodes to the goal. Because episodes reset at the start and stop after a short tail of goal self-transitions, the logged counts are not a stationary distribution of an endless reflecting walk. They need not be uniform along the corridor; the heatmap below shows the actual finite-episode coverage.

The optimal policy is trivially "walk right 59 times," so the return from the start state is $\sum_{t=0}^{58} \gamma^t \cdot r_{t+1} = \gamma^{58} \approx 0.99^{58} \approx 0.558$. (The reward is paid on arrival at the goal, after 59 steps, which is step index 58 from the start.) So we can compute every quantity exactly and ask whether each method finds it.

The printed counts show 120 of 1,680 state-action pairs observed, about 7.1%. The 60 visited cells are exactly the ones in row 3. Every other cell is unobserved, and every method below has to decide what to do there. The 7.1% figure is worth pausing on: it is not an extreme number, and yet it is enough to make a naive planner fail completely, as we are about to see.

The model is deliberately simple: a maximum-likelihood empirical transition model with a uniform fallback for unobserved pairs. This is the tabular analogue of a neural dynamics model with no regularisation, and it has the property we want to study: off-support, it returns something arbitrary, determined by our fallback choice rather than by evidence.

$$\hat P(s'\mid s,a) = \begin{cases} \dfrac{n(s,a,s')}{n(s,a)}, & n(s,a) > 0,\\[2mm] \dfrac{1}{|\mathcal{S}|}, & n(s,a) = 0, \end{cases} \qquad \hat r(s,a) = \begin{cases} \dfrac{1}{n(s,a)}\displaystyle\sum_{i:(s_i,a_i)=(s,a)} r_i, & n(s,a) > 0,\\[2mm] 0, & n(s,a) = 0. \end{cases}$$

where:

• $\hat P(s'\mid s,a)$: the estimated transition probability from $(s,a)$ to $s'$
• $\hat r(s,a)$: the estimated expected reward at $(s,a)$
• $n(s,a)$: the number of times $(s,a)$ appears in the dataset
• $n(s,a,s')$: the number of times $s'$ followed $(s,a)$
• $|\mathcal{S}|$: the number of states, used as the uniform fallback denominator

The uniform fallback is a stand-in for extrapolation, not a claim that other approximators behave identically. A neural network may continue a learned trend; a random forest prediction reflects its selected leaves; a Gaussian-process posterior mean approaches its prior mean when covariance with the training inputs vanishes, as can happen far away under a decaying kernel. None is automatically correct off-support. The uniform choice makes the failure mode vivid: an unvisited transition can land anywhere in the grid, including directly on the goal.

We use a count-based uncertainty proxy because the state-action visit counts are exact and interpretable in this tabular toy. In a larger continuous-state system, an ensemble is one common alternative; the count score isolates the penalty mechanism from neural-network training noise.

$$u(s,a) = \frac{\alpha}{\alpha + n(s,a)}, \qquad \alpha = 1.$$

where:

• $u(s,a)$: the count-based uncertainty score
• $n(s,a)$: the visit count from the dataset
• $\alpha$: the prior pseudo-count, fixed at 1 here

If a pair has never been observed, $u = 1$. If it has been observed 999 times, $u = 1/1000 = 0.001$. This is monotone, bounded in $(0,1]$, and it shrinks at roughly $1/n$ in the well-sampled regime. The $\sqrt{1/n}$ version has a different decay rate, closer to common concentration-bound scaling; here we use $1/n$ only as a simple bounded heuristic. Choosing a bounded score also keeps the penalty comparable across state-action pairs: no single pair can swamp the sum with an arbitrarily large uncertainty value.

Two things to notice. Right-action uncertainty is maximal off the corridor, because those pairs were never observed, and lower where rightward actions appear in the logs. The up and down actions remain maximally uncertain even on the corridor; that distinction is in the underlying state-action counts, though this heatmap shows only the right-action slice. A familiar state does not make every action at that state well supported.

A third thing to notice is that cell counts are not uniform even within the corridor. Fifty of the 59 nonterminal corridor cells have between 700 and 800 visits in this seeded run, but the two cells immediately before the goal have only 600 and 428. The absorbing goal has 6000 visits because every one of the 300 episodes adds a tail of 20 goal self-transitions. That bright goal cell should not be mistaken for stronger coverage of the approach states, and cell counts alone still say nothing about unsupported actions at those states.

Now we plan. The generic recipe is value iteration on the learned model:

$$\hat Q(s,a) = \hat r(s,a) + \gamma\sum_{s'}\hat P(s'\mid s,a)\hat V(s'), \qquad \hat V(s) = \max_a \hat Q(s,a),$$

where:

• $\hat Q(s,a)$: the estimated action-value at $(s,a)$ under the learned model
• $\hat V(s)$: the estimated state value, the maximum of $\hat Q$ over actions
• $\hat r(s,a)$: the estimated reward
• $\hat P(s'\mid s,a)$: the estimated transition probability
• $\gamma$: the discount factor

iterated to convergence. This is the same fixed-point computation that underlies every model-based planner in Part VII: Planning and Agency, with a learned model in place of the true one. The reason it is dangerous offline is precisely that value iteration trusts the model everywhere: the $\max_a$ operator steps through all four actions at every state, including states the model has never seen, and it takes whatever action looks best according to estimates that may be arbitrary.

The two trajectories show the mechanism. Without a penalty, the model's ignorance is read as opportunity and the policy moves off the data. With a penalty, the same model, the same dataset, and the same value iteration produce the optimal policy in this toy. A penalty of 0.5 per unobserved transition is enough to change the selected policy from zero return to the full optimum.

The previous section diagnosed the disease: an unconstrained estimate is maximised by the optimiser, and off-support the estimate is arbitrary. This section describes the cure. The idea is almost embarrassingly simple, but its theoretical consequences are substantial, and the design choices around it are where most of the engineering lives.

Let $\mathcal{M}$ be a set of MDPs consistent with the data, in the sense that no MDP in the set is ruled out by the observed transitions. Offline, the data cannot distinguish them, so the honest thing to do is to consider the worst case within the set:

$$V^{\pi}_{\text{pess}}(s) = \min_{M \in \mathcal{M}} V^{\pi}_M(s).$$

This is pessimism. It says: if the true environment lies in the candidate set, I will plan for the candidate that treats me worst. Under that condition, the pessimistic value is a lower bound on the true value of a fixed candidate policy; how informative that bound is depends on the set. The corresponding minimax formulation has the agent choose a policy, "nature" choose an MDP from the specified set, and the agent receive the worst-case value. This is a planning objective, not a guarantee of risk-free outcomes.

In practice $\mathcal{M}$ is never enumerated. Instead, the worst case is approximated by subtracting a penalty from the reward:

$$\tilde r(s,a) = \hat r(s,a) - \lambda\, u(s,a),$$

where $u$ is the uncertainty function from the previous section and $\lambda$ is the pessimism coefficient. The penalty is called "conservative" because it reduces the appeal of uncertain regions relative to their unpenalised estimates. A high predicted reward can still outweigh a high penalty; this objective does not impose a confidence threshold. The intuition is that the penalty proxies the gap between $\hat r$ and the worst-case reward over $\mathcal{M}$, and where uncertainty is large that proxy can be large.

The penalty is not automatically a guarantee. To obtain a lower bound, it must cover the model's possible overestimation at every relevant state-action pair. Here is a sufficient condition for a fixed deterministic policy.

Suppose the true and learned Bellman operators differ by at most $\epsilon$ when applied to the true value function:

$$\big|(\mathcal{T}^\pi V_M^\pi)(s) - (\hat{\mathcal{T}}^\pi V_M^\pi)(s)\big| \le \epsilon \quad \text{for all } s.$$

The contraction property then gives $\|V^\pi_M-V^\pi_{\hat M}\|_\infty\leq\epsilon/(1-\gamma)$. More locally, let the penalised model Bellman operator be $\hat{\mathcal T}_{\rm pen}^\pi V(s)=\hat r(s,\pi(s))-\lambda u(s,\pi(s))+\gamma\mathbb E_{\hat P}V(s')$. Its value, averaged over the initial distribution, is

$$\hat J_{\text{pen}}(\pi) = \hat J(\pi) - \frac{1}{1-\gamma}\,\mathbb{E}_{(s,a)\sim d^\pi_{\hat M}}\big[\lambda\,u(s,a)\big].$$

For a pointwise lower bound, a sufficient condition is

$$\lambda u(s,\pi(s))\;\geq\; (\hat{\mathcal T}^\pi V_M^\pi)(s)-(\mathcal T^\pi V_M^\pi)(s)\quad\text{for every }s.$$

Under this condition, $\hat{\mathcal T}_{\rm pen}^\pi V_M^\pi\leq V_M^\pi$. Monotonicity and contraction of the discounted Bellman operator imply

$$\hat V^\pi_{\text{pen}}(s) \le V^\pi_M(s) \quad \text{for all } s.$$

The pessimistic estimate is a certified lower bound only when the stated condition holds. Since $V_M^\pi$ and the true Bellman error are unknown offline, a count or ensemble score is not automatically a valid bound on that error. Policy-improvement guarantees require additional coverage, estimation, and optimisation assumptions; the displayed inequality alone does not give a suboptimality rate.

Two things follow. First, a penalty that is too small to cover overestimation lacks this lower-bound guarantee, whereas one that is too large can suppress useful policies. Second, a uniform penalty calibrated to a global worst case taxes even well-understood steps. Here is the deep problem: the condition depends on $V^\pi_M$, which you do not have. A generic bound is $|V^\pi_M|\leq R_{\max}/(1-\gamma)$; tighter local bounds require additional assumptions. Using only the generic bound can produce the vacuous guarantee from the worked example earlier. More selective methods restrict where pessimism enters, and the rest of this section compares those choices.

MOPO. Model-based Offline Policy Optimization learns an ensemble of probabilistic dynamics models. Its practical uncertainty score takes the largest Frobenius norm of a member's predicted Gaussian scale matrix (described in the paper as maximum standard deviation), $u(s,a)=\max_i\|\Sigma_i(s,a)\|_F$; it subtracts $\lambda u$ from reward and trains a policy with MBPO-style short model rollouts. That score is heuristic rather than the admissible model-error upper bound required by the paper's theorem. The theoretical comparison also uses the true reward function in its penalized model; the practical implementation learns a reward model. Under the theorem's function-class, exact-reward, and admissible-dynamics-error assumptions, the result compares the learned policy with any candidate policy by

$$J_M(\hat\pi) \;\geq\; \sup_\pi\big\{J_M(\pi)-2\lambda\bar u(\pi)\big\},$$

where $\bar u(\pi)$ is the discounted sum of $u(s,a)$ under policy $\pi$ in the learned model. This is a conditional guarantee, not a promise that the practical Gaussian scale (standard-deviation) heuristic upper-bounds real model error.

The gap between the theorem and the algorithm is instructive. The theorem assumes an admissible bound on relevant dynamics error and an exact reward; the practical algorithm estimates a Gaussian scale score and a reward model from data. That heuristic may work empirically, but it does not inherit the theorem's guarantee without additional calibration and reward-error assumptions. Read the proof as a statement about what the penalty would need to control, not a certification of the fitted ensemble.

MOReL. The Model-based Offline Reinforcement Learning algorithm of Kidambi and colleagues takes a harder line. It partitions the state-action space into known and unknown using ensemble disagreement, and then constructs a pessimistic MDP in which unknown pairs lead to an absorbing terminal state carrying a penalty:

$$P_p(s_{\text{halt}}\mid s,a)=1\quad\text{for unknown }(s,a),\qquad r_p(s_{\text{halt}},a)=-\kappa.$$

The halt state is absorbing and carries the negative reward at every subsequent step; known pairs use the fitted transition. Thus its value under the displayed convention is $-\kappa/(1-\gamma)$, not a one-time $-\kappa$. This hard unknown-pair rule prevents a model rollout from inventing a favorable continuation after the halt, whereas a soft penalty allows it to continue. MOReL's algorithm optionally uses a behavior-cloned policy to initialize its planner; that is not a required behavior-cloning regularizer in its objective.

COMBO. Conservative Offline Model-Based Policy Optimization learns a Q-function from a mixture of real transitions and model rollouts. Its conservative term pushes down Q-values on state-action pairs sampled under model rollouts and up Q-values on logged state-action pairs. Thus it does not apply pessimism only to real data. Unlike MOPO's practical reward penalty, this regularizer acts on the value function and does not require an explicit model-uncertainty estimate.

The COMBO idea is subtly different from the others. It does not need to know where the model is uncertain; it can simply compare where the Q-function is being trained on real data versus rollout data, and it penalises the difference. This makes COMBO less dependent on the quality of an uncertainty estimator, but more dependent on having a good critic architecture and optimiser.

RAMBO. Instead of subtracting a hand-designed uncertainty score from reward, RAMBO (Rigter and colleagues) trains the model adversarially against the value function while also fitting the data. Its model objective still has a coefficient $\lambda$ controlling the adversarial term, so pessimism has not become hyperparameter-free. It is a value-aware model-learning approach rather than an uncertainty-penalty approach. The trade-off is that the model is trained both for data fit and for appropriately pessimistic values, which adds training complexity.

It is worth knowing that the same ideas appear on the model-free side, often with different vocabulary.

• BCQ (Batch-Constrained Q-learning) trains a generative model $G_\omega(s)$ over the dataset's actions and restricts the policy to perturbing samples from it. The constraint is on the support of the action distribution.
• BEAR applies an MMD constraint between the learned policy's action distribution and the behaviour policy's during actor policy improvement; it does not add MMD to the critic's objective.
• TD3+BC combines Q-maximisation with a plain action-regression term in the actor objective: $\max_\pi\,\mathbb E_{(s,a)\sim\mathcal D}[\lambda Q(s,\pi(s))-\|\pi(s)-a\|^2]$, with $\lambda=\alpha/\mathbb E_{(s,a)\sim\mathcal D}|Q(s,a)|$. Thus $\alpha$ scales the Q term through $\lambda$, not the regression term. It is startlingly simple and competitive.
• AWAC and AWR perform advantage-weighted behaviour cloning: the policy update is a weighted maximum-likelihood fit where the weight is $\exp(A(s,a)/\beta)$. Actions with low advantage get near-zero weight, so the actor update moves toward better logged actions. That statement does not guarantee that every critic backup, including AWAC's, avoids querying actions outside the dataset.
• IQL removes the OOD action query entirely by estimating the value function with expectile regression back-ups, which never need $\max_a Q(s,a)$. It is arguably the cleanest statement of the underlying principle: do not ask the model a question the data cannot answer.

All of these address the same distribution-shift problem from different sides: pessimism discounts uncertain estimates, while a policy constraint limits which estimates are queried. They are complementary design choices, not generally a formal dual pair. The model-free family does not require a dynamics model, yet it still faces coverage problems when a critic is asked to value unsupported actions.

The implementation is one line. Everything visible in the previous section came from adding - PENALTY * u_hat to the reward.

The choice of $\lambda$ is the interesting part. Below some threshold, the penalty is too weak to overcome the model's optimism and the planner still walks off the data. Above the sampled threshold in this toy grid, the greedy policy stays on the corridor and reaches the goal; this is not a general safety guarantee. Push it much higher and you do not get a better policy, only a more pessimistic value estimate, which matters when you use that estimate to choose between candidate policies rather than to act. This is why tuning $\lambda$ is subtler than it first appears: the optimal $\lambda$ for shaping a policy may be different from the optimal $\lambda$ for evaluating one.

To see the threshold, sweep the positive values of $\lambda$ across about 3.3 orders of magnitude (with zero as an additional baseline), solve the penalised model at each value, extract the greedy policy, and evaluate that policy exactly in the true grid.

Over-conservatism deserves more than a sentence, because it is the failure mode practitioners hit most often and it does not look like failure.

In the clean gridworld above, over-conservatism is invisible in the realised return. The corridor is the only route that pays anything, so increasing the penalty leaves nonterminal behavior and realised return unchanged in this example. In a realistic environment the pattern can differ. A dataset might cover a mediocre but safe region densely and a rewarding but thinly covered region sparsely. As $\lambda$ grows, the latter pays a larger penalty even if the model's uncertainty score is bounded. The agent can retreat to the safe region and produce a stable-looking policy that misses attainable reward.

The mechanism is a tax that rises as logged coverage falls, regardless of the region's actual reward. Well-sampled regions pay almost nothing; thinly sampled regions pay much more. This biases the policy toward the well-sampled region even when the well-sampled region is not where the reward lives. The result is not a broken policy. It is a policy that looks reasonable, that runs cleanly, and that quietly underperforms.

There are three symptoms worth watching for:

• Return collapse relative to behaviour cloning. A policy that fails to beat a behaviour-cloning baseline may be over-constrained or over-penalised, though weak coverage, model error, or optimisation failure can produce the same symptom. The baseline is a useful diagnostic, not proof of the cause.
• Sensitivity of the chosen policy to $\lambda$ across the middle of the range. If the argmax policy changes shape when $\lambda$ moves from 0.3 to 0.5, the penalty is not just taxing the estimate, it is deciding the policy. That is fragile.
• A large gap between penalised and unpenalised model scores. In this fixed model, the optimal penalised score cannot increase as $\lambda$ rises. A negative penalised score shows that the objective is heavily taxed; it does not imply the task's true return is negative or that the model believes the task is worthless.

The remedy depends on the cause. Reducing $\lambda$ may recover a good policy but can also reintroduce model exploitation. Better data about the promising region, where collection is allowed, can reduce uncertainty; longer training on the same unsupported data cannot manufacture that missing evidence. A better-calibrated local penalty or constraint can distinguish well-supported and thinly supported regions. MOPO's penalty varies with $(s,a)$; MOReL halts on pairs classified as unknown. Neither automatically calibrates uncertainty to true model error.

An optimiser can seek out overestimated actions, so upper-tail value errors are especially dangerous. Underestimation also matters: it can discard a good policy or prevent improvement beyond the behaviour policy. Conservative methods trade some of that opportunity for protection against unsupported optimistic choices. Whether the trade is appropriate depends on coverage and the decision at hand.

Pessimism and constraints are two responses to distribution shift, often used together rather than as formal duals. This section covers the constraint family, then turns to a separate problem: how can you judge a learned policy without running it?

Pessimism reduces estimated values where a penalty signals model risk, seeking to discourage unsupported choices. A support constraint instead restricts the policies considered. Neither automatically ensures that the resulting policy stays near the data: a heuristic penalty can miss model error, and an approximate constraint can admit poorly covered actions. They act on different parts of the optimization problem, and their effectiveness depends on their assumptions and implementation.

A support constraint restricts $\pi$ to the set of policies whose state-action distribution is covered by the dataset:

$$\pi \in \Pi_{\text{supp}} = \Big\{ \pi : d^\pi(s,a) > 0 \Rightarrow \rho_\beta(s,a) > 0 \Big\}.$$

This is an idealized constraint on the full state-action occupancy, including states reached by the target policy. BCQ's conditional generative model and bounded action perturbation approximate the behavior-action support at queried states; they do not enforce this global occupancy condition, especially at unseen states. Under a continuous-density behaviour distribution, each individual sampled point has probability zero even when the distribution has full-dimensional support. That conditional support is hard to infer from a finite log, so practical methods use divergences or generative action models as approximations.

A behaviour constraint is softer: rather than requiring full support, it penalises divergence from the behaviour policy:

$$\max_\pi \; \mathbb{E}_{s\sim\mathcal{D}}\Big[\mathbb{E}_{a\sim\pi(\cdot\mid s)}Q(s,a) \;-\; \beta\,D\big(\pi(\cdot\mid s)\,\|\,\pi_\beta(\cdot\mid s)\big)\Big],$$

where $D$ denotes a discrepancy or penalty, such as KL, MMD, or squared action distance. BEAR constrains its actor with MMD, while TD3+BC uses an action-regression term. AWAC instead performs advantage-weighted behaviour cloning; its update is not literally this displayed divergence objective. The choice matters: KL is asymmetric, MMD compares distributions through a kernel, and squared distance ignores much of the action distribution's structure.

These mechanisms can be complementary, but no general equivalence follows. A support constraint may still allow optimistic errors within its permitted set; a pessimistic critic may still be poorly calibrated where data are scarce. CQL applies conservatism to critic values, MOPO penalises rewards in model rollouts, and MOReL changes unknown transitions to lead to HALT. BEAR and TD3+BC illustrate direct actor restrictions. Specific implementations may combine mechanisms, but the papers do not establish a universal constraint–penalty duality.