Traditional collision avoidance algorithms and formation control methods face numerous challenges when dealing with complex environments. To address this, a method based on Deep Reinforcement Learning (DRL) and trajectory optimization is proposed to achieve multi-robot navigation and formation maintenance. By combining the powerful learning ability of DRL with the precision of trajectory optimization, this method addresses the issue of obstacle avoidance and navigation for multi-robots in complex environments while maintaining formation. Meanwhile, a differentiable formation metric based on graph theory is defined and integrated with DRL, leading to the proposal of a novel Proximal Policy Optimization combined with Trajectory Optimization (PPOTO) algorithm. Experimental results demonstrate that this method performs excellently in multi-robot navigation and formation maintenance tasks, achieving significant performance improvements compared to traditional methods.