The precision of absolute navigation data and the accuracy of relative positioning within a UAV (Unmanned Aerial Vehicle) swarm are critical determinants for the successful execution of coordinated group missions. In regions where satellite signals are obstructed, UAVs are unable to acquire precise absolute positional data, which adversely affects the maintenance of both absolute and relative positioning accuracy within the UAV swarm. To address this challenge, a hybrid cooperative navigation method based on range optimization and state transformation extended Kalman filter (ST-EKF) is proposed. This method optimizes the UAV’s position information by incorporating both its own position data and relative range measurements from other UAVs. Furthermore, to enhance navigation accuracy, a fusion approach based on ST-EKF is employed to estimate and correct navigation errors for each UAV. Simulation experiments demonstrate that the proposed hybrid cooperative navigation method (Hybrid-CN) outperforms the conventional cooperative navigation method based on the extended Kalman filter (EKF-CN) in terms of performance. Specifically, when utilizing the Hybrid-CN method for navigation solutions, the relative position error within the UAV swarm is decreased by 41.7%, the positional Root Mean Square Error (RMSE) has been diminished by 28.8%, and the velocity RMSE has been diminished by 29.3% relative to the EKF-CN methodologies.