The paper devotes to studying the formation tracking control of the UAV cluster system when external disturbance is present, which uses switching topology to depict the communication connectivity among each UAV. The leader-follower approach is introduced to tackle the formation issue. The controller is designed by combining the disturbance observer and the consistency theorem, and then the formation tracking algorithm is constructed, the controller’s parameters are acquired through solving the linear matrix inequality. Besides, the segmented Lyapunov function is constructed to complete the stability proof part of the theoretical derivation. Then numerical simulation experiments based on Matlab are carried out, taking a cluster system of 6 UAVs as an example, the anti-disturbance performance and stability of the system are investigated under two situations: without adding a disturbance observer and with adding a disturbance observer. The number of UAVs is augmented to 11 for the purpose of evaluating the scalability.