Particle Swarm Optimization (PSO) performance relies heavily on interaction topology. Our prior work, Self-Evaluated Topology Particle Swarm Optimization (SET-PSO), achieved high accuracy and stability through self-evaluated dynamic topology, but suffered from significant computational inefficiency due to global topological changes and multi-run evaluations. To solve this problem, this paper proposes the Grouping and Mediator Self-Evaluating Topology Particle Swarm Optimization (GMSET-PSO). GMSET-PSO incorporates a novel Grouping and Mediator model, organizing particles into localized Von Neumann groups with virtual mediator particles facilitating hierarchical, inter-group communication. This design effectively blocks global information propagation, localizing search and reducing computational burden. Furthermore, dynamic topology adaptation becomes group-specific, and topology evaluation is streamlined to a single-run assessment with immediate rollback if no improvement. On the CEC2020 and CEC2022 benchmark function set, GMSET-PSO was compared with SET-PSO and other PSO variants equipped with adaptive mechanisms. The results show that GMSET-PSO largely retains the accuracy of the original algorithm while improving stability by 46.08% and efficiency by 25.90%. Moreover, it significantly outperforms the other algorithms, particularly in solving more complex problems. Showcasing a significant advancement in scalable and efficient self-adaptive optimization.