Infrared thermal imaging compensates for the shortcomings of the traditional visual camera in rain, snow, and other low visibility environments, provides all-weather and efficient environment sensing capability for automatic driving vehicles, and improves the safety and reliability of the automatic driving system. With the popularization of intelligent driving, physical adversarial attacks also pose a severe threat to DNN-based infrared object detectors. However, current research on adversarial attacks against infrared thermal imaging primarily concentrates on pedestrian detection. These methods focus on locating the bounding box of the attacking object and then generating the optimal adversarial patch within it. When these methods are applied directly to vehicle adversarial attacks, the adversarial patch often extends beyond the vehicle’s body, making it challenging to execute physical attacks. To address this issue, we propose a two-stage optimized adversarial patch generation method for physically attacking infrared image vehicle detectors, called TSAP. TSAP employs a two-stage optimization process: firstly, the target vehicle is segmented from the image using the DeepLab-v3+ model to ensure that the patch fits better with the target vehicle for subsequent physical implementation; secondly, the physical parameters of the patch are optimized using the Differential Evolution (DE) algorithm. Furthermore, we design a novel loss function to guide the optimization of the adversarial patches, and this improvement effectively increases the Attack Success Rate (ASR) and enables multi-target attacks. After extensive experiments in digital and physical environments, we verify the proposed method’s effectiveness and robustness. In addition, the process has better convergence due to the reduced global search space after segmentation. Since the method can perform a simple but effective physical attack on infrared thermal imaging vehicle detectors, it poses a potential threat to the widespread use of autonomous driving in the real world.