Tracing the Roots: Trust Region Policy Optimization

This report examines the origins of the celebrated Trust Region Policy Optimization (TRPO) framework, an application of natural gradient methods in reinforcement learning. While the standard objective of an agent is to maximize cumulative reward, traditional policy gradient approaches are often highly sensitive to step size selection because gradients are typically computed in the parameter space rather than the actual underlying policy space. TRPO addresses this instability by introducing a ‘trust region’ constraint based on the Kullback-Leibler divergence between different policies. This ensures that each policy update remains within a safe distance of the previous iteration. By leveraging the Fisher Information Matrix as an invariant metric to guide the natural gradient, the framework provides a robust optimization strategy that guarantees monotonic improvement of the policy.

Unpublished