Efficient exploration and policy diversity optimization remain pivotal challenges in deep reinforcement learning (DRL). A common approach is to maintain a population of agents, each optimized simultaneously to achieve high task-returns while exhibiting diverse behaviors. However, such methods necessitate multiple parallel training chains with frequent inter-chain interactions, leading to redundant initialization and inefficiencies in time and memory usage. To address these limitations, we propose the Stein Self-Repulsive Proximal Policy Optimization (SRPPO) framework, which generates diverse policies within a single-process Markov decision chain. SRPPO extends the standard policy gradient method by integrating a self-repulsion objective based on Stein variational policy gradient. This objective drives the generation of new policy away from the historical policy ensemble, thereby promoting global exploration and policy diversity. Moreover, SRPPO enriches the design of self-repulsion by incorporating the behavioral distribution characteristics of agents. This approach allows for the progressive discovery of multiple high-quality policies over time without falling into local optima traps common with incremental updates. Experimental results on various continuous control tasks show that SRPPO effectively generates diverse skills, avoids deceptive rewards, and improves exploration efficiency.