Offline and Conservative Model-Based RL › Learning from Fixed Datasets

This section fixes notation, states a coverage assumption that can support offline learning, and derives the simulation lemma, which is the single most useful equation in the chapter. It explains why a model that looks accurate can produce a badly wrong value function, and it draws the line between model-based and model-free approaches to the offline problem. Later we will distinguish coverage from justified structural assumptions that can also constrain unobserved outcomes.

We have a Markov decision process $M = (\mathcal{S}, \mathcal{A}, P, r, \gamma, d_0)$ as introduced in Part II: Inference, Decisions, and Control. To keep the comparison with learned models clean, we will assume throughout that the state is fully observed and the MDP is stationary. Neither assumption is free. Under partial observability the effective state is a belief, and the coverage problem becomes harder because two observation histories can be indistinguishable while requiring different actions. We will flag where that matters.

The stationarity assumption matters too. If the environment drifts over time, last year's transitions may not describe the world the deployed policy will face. In the warehouse example, a reorganised floor plan changes the dynamics represented by the logs. Here we assume no such drift: training and deployment share the same environment.

Instead of interacting, we are handed a dataset

$$\mathcal{D} = \{(s_i, a_i, r_i, s_i')\}_{i=1}^{N}, \qquad (s_i, a_i) \sim \rho_\beta, \quad \mathbb{E}[r_i\mid s_i,a_i]=r(s_i,a_i), \quad s_i' \sim P(\cdot \mid s_i, a_i),$$

where:

• $\mathcal{D}$: the fixed dataset of $N$ transitions
• $s_i$: the $i$-th observed state
• $a_i$: the $i$-th observed action taken in that state
• $r_i$: the observed reward, whose conditional mean is $r(s_i,a_i)$
• $s_i'$: the next state, drawn from the transition distribution $P(\cdot\mid s_i, a_i)$
• $\rho_\beta$: the behaviour distribution, the state-action distribution induced by whatever process generated the logs

The behaviour distribution might come from a human demonstrator, a hand-tuned controller, a previously deployed policy, or five such policies mixed together. We do not need to know which. We only need the samples. Notice that the dataset records actions and their outcomes but never explains them. Nothing tells us why the behaviour policy chose a given action, which matters later when we want to imitate it in states where the model is uncertain.

The objective is unchanged from online RL. We want a policy $\pi$ maximising

$$J(\pi) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \;\middle|\; s_0 \sim d_0,\; a_t \sim \pi(\cdot\mid s_t),\; s_{t+1}\sim P(\cdot\mid s_t, a_t)\right].$$

where:

• $J(\pi)$: the expected discounted return of policy $\pi$
• $\gamma$: the discount factor in $[0,1)$
• $r(s_t, a_t)$: the reward received at step $t$
• $d_0$: the initial-state distribution, with $s_0\sim d_0$
• $\pi(\cdot\mid s_t)$: the policy, giving the action distribution at state $s_t$
• $P(\cdot\mid s_t, a_t)$: the environment transition distribution

The difference from online RL is entirely in what we are allowed to do while searching for $\pi$. Online, we can test $\pi$, observe the return, and adjust. Offline, that loop is severed: the objective is still defined against the true environment, but we can only ever evaluate candidate policies against our beliefs about it.

It is convenient to work with discounted state-action occupancy measures rather than raw trajectories. For a policy $\pi$ with initial distribution $d_0$, define the mass of any measurable state-action set $B$ by

$$d^\pi(B) = (1-\gamma)\sum_{t=0}^{\infty} \gamma^{t}\, \Pr((s_t,a_t)\in B \mid \pi, d_0).$$

where:

• $d^\pi(B)$: the discounted occupancy mass that policy $\pi$ assigns to set $B$
• $\Pr((s_t,a_t)\in B\mid\pi, d_0)$: the probability of visiting $B$ at step $t$ under policy $\pi$ starting from $d_0$
• $1-\gamma$: the normalising factor that makes the sum a probability distribution

The $(1-\gamma)$ factor makes $d^\pi$ a probability distribution over $\mathcal{S}\times\mathcal{A}$. In a discrete space, $d^\pi(s,a)$ abbreviates the mass of the singleton $\{(s,a)\}$; in a continuous space, it denotes a density only when one exists with respect to a stated reference measure. Why bother with this object rather than working with the trajectory distribution directly? Because an occupancy measure summarises "where the policy spends its time," weighting near-term visits more heavily than far-future ones. It converts the trajectory of the policy into a static picture of the space, which is exactly what we need to compare two policies and to reason about how much their visited regions overlap. With this normalisation, the objective is a simple expectation:

$$J(\pi) = \frac{1}{1-\gamma}\,\mathbb{E}_{(s,a)\sim d^\pi}\big[r(s,a)\big].$$

Now compare $d^\pi$ with $\rho_\beta$. If a policy spends time where the behavior distribution has substantial mass, the data can inform those choices; where it has none, learning requires additional structure. For the displayed ratio, read both quantities as masses in a discrete space or densities with respect to a common reference measure in a continuous space. This intuition is captured by the concentrability coefficient:

$$C(\pi) \;=\; \sup_{s,a} \; \frac{d^\pi(s,a)}{\rho_\beta(s,a)} .$$

where:

• $C(\pi)$: the concentrability coefficient of policy $\pi$
• $d^\pi(s,a)$: the discounted occupancy of $(s,a)$ under policy $\pi$
• $\rho_\beta(s,a)$: the behaviour distribution's density at $(s,a)$
• $\sup_{s,a}$: the supremum over state-action pairs in the discrete case, or the essential supremum of the density ratio in the continuous case

Define the ratio as infinite when the target occupancy assigns positive mass to a region where the behavior distribution assigns zero; zero-over-zero pairs do not affect it. In continuous spaces, this requires absolute continuity and an essentially bounded density ratio. Thus finite $C(\pi)$ requires the occupancy of that policy to be supported by the behavior distribution. Some offline-RL guarantees assume coverage of an optimal policy, $C^\star=C(\pi^\star)<\infty$, together with further statistical and algorithmic conditions. Covering one optimal policy is weaker than covering every policy, but it does not by itself make any particular learning algorithm successful. A small coefficient says the behavior distribution gives reasonable weight to what the policy visits; a large one signals weak coverage.

Coverage is one way to make offline learning identifiable. Without coverage or justified structure linking unobserved outcomes to observed ones, no method can guarantee good decisions uniformly over the unrestricted MDP class.

The coefficient compares state-action occupancy densities. The next picture simplifies them to their state marginals: if the target visits states the behaviour never reaches, then it necessarily visits unsupported state-action pairs and $C(\pi)$ is infinite. A gap confined to actions at otherwise covered states would also matter, but this state-only picture does not show it.

Now suppose we learn a model $\hat M = (\hat P, \hat r)$ from $\mathcal{D}$ and use it in place of the environment. How wrong is the resulting value function? The answer is the simulation lemma, and it deserves a careful derivation because it is the theoretical heart of the whole chapter.

Fix a deterministic policy $\pi$ for this derivation; a stochastic policy gives the same identity after averaging each action-dependent term over $a\sim\pi(\cdot\mid s)$. Write $V^\pi_M$ for its value under the true MDP and $V^\pi_{\hat M}$ for its value under the model. Define the gap $\Delta(s) = V^\pi_M(s) - V^\pi_{\hat M}(s)$. Expanding both Bellman equations,

$$\begin{aligned} V^\pi_M(s) &= r(s,\pi(s)) + \gamma\, \mathbb{E}_{s'\sim P(\cdot\mid s,\pi(s))}\big[V^\pi_M(s')\big],\\ V^\pi_{\hat M}(s) &= \hat r(s,\pi(s)) + \gamma\, \mathbb{E}_{s'\sim \hat P(\cdot\mid s,\pi(s))}\big[V^\pi_{\hat M}(s')\big]. \end{aligned}$$

Subtracting the second from the first and adding and subtracting a term that evaluates $V^\pi_M$ under the model,

$$\Delta(s) = \underbrace{\big(r - \hat r\big)}_{\text{reward error}} + \gamma \underbrace{\Big(\mathbb{E}_{P} V^\pi_M - \mathbb{E}_{\hat P}\, V^\pi_M\Big)}_{\text{transition error, evaluated on the true value}} + \gamma\, \mathbb{E}_{\hat P}\big[\Delta(s')\big].$$

where:

• $\Delta(s) = V^\pi_M(s) - V^\pi_{\hat M}(s)$: the value gap at state $s$
• $r - \hat r$: the reward-model error, abbreviated $r(s,\pi(s)) - \hat r(s,\pi(s))$
• $\mathbb{E}_P V^\pi_M - \mathbb{E}_{\hat P}\, V^\pi_M$: the transition error, evaluated using the true value function $V^\pi_M$
• $\mathbb{E}_{\hat P}[\Delta(s')]$: the expected continuation of the gap under the model's dynamics

The adding-and-subtracting trick is the algebraic seed of the whole result. We insert $\mathbb{E}_P V^\pi_M - \mathbb{E}_{\hat P} V^\pi_M$ not because it was there already, but to isolate the reward mismatch and the transition mismatch as a single "local inconsistency" while leaving a clean recursion for the remaining gap. Everything that follows is a matter of chasing where that local inconsistency gets weighted.

The first two terms do not depend on $\Delta$; call their sum the local inconsistency $\ell(s)$. The recursion $\Delta = \ell + \gamma \mathbb{E}_{\hat P}\Delta$ unrolls along the trajectory distribution generated by the model. Write $d^\pi_{\hat M}(s,a)$ for the model's normalised discounted state-action occupancy and $d^\pi_{\hat M}(s)$ for its state marginal, both induced by the learned model $\hat M$ rather than the true MDP. Averaging the initial-state gap over $s_0\sim d_0$ gives

$$\boxed{\;J_M(\pi)-J_{\hat M}(\pi) \;=\; \frac{1}{1-\gamma}\,\mathbb{E}_{s\sim d^\pi_{\hat M}}\big[r(s,\pi(s)) - \hat r(s,\pi(s)) + \gamma\delta(s)\big]\;}$$

Here $J_M(\pi)=\mathbb E_{s_0\sim d_0}V_M^\pi(s_0)$, $J_{\hat M}(\pi)=\mathbb E_{s_0\sim d_0}V_{\hat M}^\pi(s_0)$, and $\delta(s)=\mathbb{E}_{P(\cdot\mid s,\pi(s))}[V^\pi_M(s')] - \mathbb{E}_{\hat P(\cdot\mid s,\pi(s))}[V^\pi_M(s')]$. A pointwise value-gap identity would instead use a model occupancy conditioned on that particular starting state; the displayed occupancy is averaged over $d_0$.

The boxed equation averages local errors over states visited by a fixed policy in the learned model, not necessarily in reality. If the model's dynamics place substantial occupancy in a region where its predictions are wrong, that region contributes heavily to the value gap. The identity does not say that error itself attracts a fixed policy toward that region.

Error compounding and model exploitation are distinct steps. The fixed-policy identity describes how one-step errors propagate through model rollouts. When a policy is then selected by maximizing learned-model value, it may favor optimistic, poorly supported regions; that selection can change the model occupancy in the identity. The two mechanisms can reinforce each other, but neither the identity nor a fixed-policy bound alone proves that the errors concentrate exactly where they are largest.

Bound the two error terms to get a usable inequality. Suppose the reward model is off by at most $\epsilon_r$ and the transition model has total-variation error at most $\epsilon_P$:

$$\big|r(s,a) - \hat r(s,a)\big| \le \epsilon_r, \qquad \big\|P(\cdot\mid s,a) - \hat P(\cdot\mid s,a)\big\|_{TV} \le \epsilon_P .$$

where $\epsilon_r$ bounds the reward error, $\epsilon_P$ bounds the total-variation transition error, and $R_{\max}$ is the maximum absolute reward. Since $|V^\pi_M(s')| \le R_{\max}/(1-\gamma)$ and $|\mathbb{E}_p f - \mathbb{E}_q f| \le 2\|p-q\|_{TV}\max|f|$,

$$\big|V^\pi_M(s) - V^\pi_{\hat M}(s)\big| \;\le\; \frac{\epsilon_r}{1-\gamma} \;+\; \frac{2\gamma R_{\max}\,\epsilon_P}{(1-\gamma)^2}.$$

where:

• $\epsilon_r$: the maximum reward-model error, $\epsilon_r \ge |r(s,a) - \hat r(s,a)|$
• $\epsilon_P$: the maximum total-variation transition-model error
• $R_{\max}$: the maximum absolute per-step reward
• $\gamma$: the discount factor

The $1/(1-\gamma)$ and $1/(1-\gamma)^2$ prefactors come from two separate sources, and it is instructive to see both. The reward error is a per-step bias paid every step, so summing it under discounting costs the horizon factor $1/(1-\gamma)$. The transition error is more expensive: total-variation error is compared against value functions bounded by $R_{\max}/(1-\gamma)$, and each step of the horizon multiplies the accumulated value bound by another factor, so the sum produces $1/(1-\gamma)^2$. The quadratic blow-up in the transition term is exactly the compounding effect, and it is the term that decides whether the bound is usable.

Now plug in an illustrative set of numbers. Take $R_{\max}=1$ and $\gamma = 0.99$. If rewards lie in $[0,1]$, then returns lie in $[0,100]$; under the weaker $|r|\leq 1$ assumption, returns lie in $[-100,100]$. Let the transition model have a uniform total-variation error upper bound of $\epsilon_P = 0.05$ per state-action pair. This is a distance between distributions, not a claim that 95% of predicted transitions are correct. The transition-error term gives

$$\frac{2 \times 0.99 \times 1 \times 0.05}{(1-0.99)^2} \;=\; \frac{0.099}{10^{-4}} \;=\; 990,$$

With $\epsilon_r=0.01$, the reward-error term adds 1, so the full bound is 991. Even the transition term alone exceeds a range-based upper bound of 201 on the value difference when true rewards satisfy $|r|\leq 1$ and the learned reward may differ by up to $\epsilon_r=0.01$. The bound is therefore vacuous in this example: a uniform 0.05 per-step TV bound does not yield a useful guarantee at this discount factor.

Plotting the two error terms against the discount factor shows which one drives the blow-up.