Offline and Conservative Model-Based RL › Summary

During its offline training phase, offline RL learns from a fixed dataset without further interaction. The removal of the training-time feedback loop creates two coupled problems: the log supplies no direct target where it is silent, and the optimiser can favour those places when unsupported model estimates are optimistic. Justified structural assumptions may narrow the uncertainty, but the log alone cannot test those predictions there.

• Coverage or justified structure is fundamental. Without sufficient logged coverage or informative assumptions that constrain unobserved outcomes, uniform worst-case improvement is impossible. A finite concentrability coefficient formalizes one form of coverage; different algorithms use different structural and statistical assumptions.
• The simulation lemma explains fixed-policy error. The value gap is an average of local errors weighted by the policy's learned-model rollout occupancy, which can differ from its true occupancy. Policy optimization can then favor optimistic model estimates and change where that learned-model occupancy lies. Error propagation and policy selection can reinforce each other, but the fixed-policy identity alone does not prove exploitation.
• Worst-case bounds can be vacuous. In the illustrative example with $\gamma=0.99$ and a uniform 0.05 transition-TV bound, the transition-error term is 990 and the full bound, including $\epsilon_r=0.01$, is 991. Both exceed a range-based upper bound on the possible value difference under the stated reward range. Useful guarantees need stronger information about where the policy goes and where the model is accurate.
• Uncertainty scores need interpretation. Counts and ensemble disagreement can flag missing evidence, but predictive variance may also include irreducible randomness. Shared model biases can make an ensemble agree while all its members are wrong.
• Pessimism can lower-bound value under conditions. A penalty must cover possible model overestimation at relevant pairs; a heuristic score alone is not a certificate. Too little penalty permits exploitation, while too much can discard useful policies.
• Constraints are complementary. Support constraints and behaviour regularisers restrict where a policy may go; pessimism reduces the estimated worth of uncertain choices. Some methods combine these mechanisms, but they are not generally formal duals.
• Evaluation remains difficult. Trajectory and per-decision importance sampling can be unbiased under support, sampling, and complete-return or valid tail-correction conditions, but may have severe variance under policy mismatch. Weighted IS can introduce finite-sample bias; fitted Q-evaluation and model rollouts introduce approximation or model bias. Coverage diagnostics help identify risk but do not certify an estimate.
• Model selection is a separate problem. Choosing among candidate policies and hyperparameters using offline data only needs its own evaluation and uncertainty checks; a favorable learned-model estimate is not enough.

The chapter's gridworld compressed the argument into a few hundred lines of NumPy: one row of a seven-row grid, one behaviour policy that never left it, and an additive reward penalty that changed the selected policy from zero return to the true optimum. The general mechanism is model exploitation: an optimiser can favor actions where estimates are unsupported. Beyond the toy grid, pessimism helps only to the extent that its uncertainty score tracks the relevant errors.