With the increasing development of Internet of Things (IoT) technology, the Unmanned Aerial Vehicle (UAV) cluster has demonstrated its advantage in numerous applications especially when performing the air-to-ground object detection missions. In the object detection missions, by Machine Learning (ML) method to make the accurate object detection decisions. However, the training of traditional ML models typically relies on the model infrastructures with high computational complexity, and UAVs are often forced to wait for the model gradients or parameters which are transmitted by the base station, thus resulting in the long waiting time of object detection missions. In addition, UAVs must transmit massive data to the base station for the centralized training, which gives rise to the high communication overload. Moreover, UAV cluster still faces some potential threats (e.g., lightning strikes, cyber-attacks, severe weathers), which could destroy some UAVs and the base station, and especially when the base station is destroyed, the object detection missions of UAV cluster fail undoubtedly. To this end, we propose a Federated Learning (FL) framework with Peer-to-Peer (P2P) architecture. In this framework, each UAV is equipped with part of the FL model and undertakes the auxiliary functions of completing the forward propagation and back propagation to realize the parallel and distributed training. We also group the UAVs with the goal of maximizing the group uniformity and apply a flexible group leader selection method, thus enhancing the resistance of UAV cluster against the potential destruction of some UAVs. We have conducted extensive experiments to prove that our framework can ensure the high detection accuracy, enhance the resistance of UAV cluster, and shorten the mission latency.