ERL-CDR: Evolutionary Reinforcement Learning with Causal Decoupling Representation
摘要
Evolutionary Reinforcement learning (ERL) algorithms effectively address the sample efficiency problem in continuous state-action spaces by integrating gradient-based reinforcement learning (RL) with gradient-free evolutionary algorithms (EAs). However, traditional ERL methods still face limitations, including insufficient processing of the state space in collected samples, diversity-limited fitness evaluation based solely on rewards leading to reduced sample diversity, and differences in EA-RL sample distributions that hinder the value function learning. This paper proposes Evolutionary Reinforcement Learning - Causal Decoupling Representation (ERL-CDR). which addresses the issue of insufficient state space processing by introducing a causal representation learning mechanism that disentangles task-irrelevant features from decision-critical features. A behavioral diversity mechanism is used to evaluate the population, promoting more comprehensive exploration of the sample space and increasing sample diversity. Finally, a novel value function is introduced to better represent the sample space of policy populations, which, together with the RL value function, effectively guides policy learning. Experimental results show that this method significantly improves convergence speed, stability, and final performance in complex continuous control tasks. These methods are of research significance for complex continuous reinforcement learning tasks.