Vision-based object detection in unmanned aerial vehicles (UAVs) is crucial for autonomous navigation and re-mote sensing. However, reliance on deep neural networks makes UAVs vulnerable to adversarial attacks, where adversarial patches can exploit model sensitivity and cause recognition errors or omissions, threatening UAV safety. Current studies focus on digital optimization in controlled labs but overlook real-world physical factors like motion blur, lens quality, and weather disturbances. We propose the Data-Augmentation Robust Adversarial Patch (DARAP) to address this. The DARAP incorporates a unified environmental-perturbation simulation module. Un-like common stochastic-noise methods, this module utilizes a rigorous mathematical framework to model diverse environmental factors. It enhances the adversarial training data, thereby improving the patch robustness in complex environments. Furthermore, it includes a feature-sensitivity-driven localization mechanism integrated with Grad-Cam to optimize patch placement. Experiments show DARAP achieves higher attack success rates compared to conventional methods in a complex and disturbed physical operating environment. This work provides valuable insights for improving adversarial attack robustness in complex environments.