In multi-agent reinforcement learning (MARL), agents continually interact with the environment, accumulating experiences and learning through trial and error. However, this prevailing approach often treats all experiences equally, neglecting the differential impacts that successful and unsuccessful experiences can have on the learning process. To alleviate this problem, we propose GATE, a guided contrastive state space method designed to utilize existing experiences more effectively. GATE introduces a novel contrastive approach that encourages agents to move towards superior states while avoiding inferior ones. It also implements a new intrinsic reward mechanism to guide agents’ exploration and exploitation processes. Moreover, GATE can be integrated as a plugin into various reinforcement learning methods. Experimental results across different settings demonstrate the effectiveness of GATE compared to other methods.