Unlike the limited field of view of a single unmanned aerial vehicle (UAV), multiple UAVs can collaboratively track targets from various perspectives. In such cases, the distance control between UAVs and the dynamic target is critical. A proper following distance provides additional response time to adjust to the target’s sudden changes. In this work, we propose an effective distance-keeping trajectory planning method for UAV swarm target tracking. First, the A* algorithm is employed to achieve the front-end path search in the grid map. Then the trajectory is optimized through a constructed cost function considering the constraints of dynamics feasibility, collision-free, swarm reciprocal distance, and target tracking distance. Notably, we not only optimize the distance between UAVs and the target but also the state of the endpoint planned for UAVs. The effectiveness of the proposed method is verified in a high-fidelity simulator. As a result, the framework proposed could track the target in complex environments from multi-view and keep a proper distance.