Robot Learning

TRPO (Trust Region Policy Optimization)

TRPO (Trust Region Policy Optimization) is an on-policy reinforcement learning algorithm introduced by Schulman et al. in 2015 that stabilizes policy-gradient training by constraining each update so the new policy stays within a KL-divergence trust region of the old one, guaranteeing approximate monotonic improvement. It solves a constrained optimization using conjugate gradients and was a milestone for continuous-control learning, though its second-order machinery was largely superseded by the simpler first-order PPO.

Why it matters for physical AI

Trust-region thinking, keeping policy updates conservative to avoid catastrophic performance collapse, underlies the PPO-family algorithms that train today's locomotion policies and increasingly fine-tune robot foundation models with reinforcement learning.