Offline-to-online reinforcement learning with policy ensemble and policy-extended value
摘要
Offline-to-online reinforcement learning, which combines the benefits of offline pretraining and online fine-tuning, promises enhanced sample efficiency and policy performance. However, existing methods, effective as they are, suffer from suboptimal performance and limited adaptability. We propose a novel framework, PEPeV, which overcomes the aforementioned limitations by leveraging the Policy Ensemble and Policy-extended Value. In the offline stage, we leverage a Behavior Cloning (BC) + Behavior Proximal Policy Optimization (BPPO) ensemble to train a set of diverse policies, while simultaneously training a policy-extended state-value and Q-value functions for seamless transition from offline to online learning and better generalization. In the online stage, we adopt the Ensemble Proximal Policy Optimization (EPPO) algorithm with policy-extended values to improve fine-tuning performance. Extensive experiments on locomotion, navigation and hand tasks demonstrate that PEPeV achieves competitive performance with SOTA baselines, and ablation studies confirm that the policy ensemble contributes +21.2% in online performance, while the policy-extended value yields an additional +1.1%. These results demonstrate PEPeV’s effectiveness in enhancing online policy quality.