Offline and Conservative Model-Based RL › Limitations and Impact

Pessimism can reduce an optimiser's incentive to exploit model error where the penalty captures the relevant risk. The gridworld made the mechanism visible: the same fitted model, dataset, and value iteration produced a zero-return policy without a penalty and an optimal policy with a reward penalty of half a unit per under-supported transition. That is an instructive example, not a universal guarantee that pessimism resolves every ill-posed offline decision problem.

Pessimism also does not manufacture information about unvisited actions. If the log contains no evidence about an action and no justified structural assumption links it to observed actions, two environments can agree on every logged transition yet require different decisions there. That rules out a uniform worst-case guarantee from the log alone. Known physics, smoothness, or transferable pretraining may provide additional information, but their assumptions and target-domain validity must be made explicit. A warehouse fleet that has only driven down the middle of the aisles has no logged outcomes from the edges. A conservative policy may avoid an edge route that is actually useful; its caution is a response to uncertainty, not proof the route is bad. Where feasible, collect relevant data; otherwise state what prior structure supports any extrapolation and how it was checked.

The practical costs are real. Pessimism introduces at least one task-dependent hyperparameter. In our gridworld policy quality changes abruptly across a threshold in $\lambda$; real problems can change less cleanly, and penalties interact with rollout length, model capacity, ensemble design, and reward scale. Training multiple models increases modelling cost. Because the dataset is fixed, additional fitting or ensemble members cannot by themselves supply transition evidence in an unsupported region; new data would be required to resolve that uncertainty empirically.

There is also a subtler limitation: ensemble disagreement need not reflect shared model bias. Members trained on the same data and similar architectures may agree on a wrong extrapolation, leaving a small disagreement score where the actual transition error is large. Adding similar members does not reliably resolve that missing-data problem. Broader pretraining may help in some settings, a direction discussed in Part IX: Foundation and World-Action Models, but its uncertainty must still be checked against the target domain.

Under-dispersion limits a pessimistic method that relies on that uncertainty score, not offline RL as a whole. If the score is systematically small where model error matters, the penalty is small there too, so it cannot be treated as a certified error bound. More varied prior data or model assumptions might help, but neither certifies calibrated uncertainty without target-domain checks.

The offline setting forces coverage, model validity, and estimation bias into the open because there is no new interaction to check a candidate policy during learning. Online feedback can sometimes expose errors, but it does not automatically rescue a poor policy or make exploration safe. The same discipline matters beyond offline RL: in the evaluation protocols of Part XI: Evaluation and Understanding, in the safety arguments of Part XII: Reliable World Models, and wherever a learned model informs a decision before its consequences can be checked.