Classical Planning and Optimal Control › Key Parameters

The key parameters for the planning methods in this chapter are:

• gamma ($\gamma$): The discount factor used in value iteration, set to $0.95$. For bounded discounted problems, it is the Bellman operator's worst-case contraction modulus. Values close to one put more weight on distant rewards and give a slower worst-case convergence bound.
• tolerance: The sweep-stopping threshold, set to 1e-6 on the maximum update. It is a residual threshold; the contraction bound converts it to a value-error bound by dividing by $1-\gamma$.
• Q, R: The state and control cost matrices for LQR, set to $Q = I$ and $R = 0.5I$. Their relative weighting changes the feedback gain and the trade-off between state regulation and control effort; the response need not be a simple proportional scaling.
• u_max: The normalized angular-acceleration input bound for pendulum swing-up, set to 2.0, below the gravity coefficient at horizontal. The optimized solution uses momentum to reach the target under this bound.
• lr: The Adam learning rate, set to 0.05. A value that is too large can make parameter updates unstable, while a very small value may converge slowly. Its useful scale depends on the horizon, dynamics, parameterization, and objective.