Small Unmanned Aerial Vehicles (UAVs) are widely used in a broad range of civilian and military applications. UAV models exhibit intricate nonlinear characteristics, significant coupling between longitudinal and lateral motions, and a susceptibility to model disturbances. In light of these complexities, it is essential to devise a flight control scheme that is robust and optimized, addressing the unique challenges posed by UAVs’ diverse missions and nonlinear dynamics. Nonlinear Dynamic Inversion (NDI) is one of the control solutions for UAV’s flight control system as it can efficiently decouples the model and successfully handle the system’s nonlinearity. Besides, the incremental approach of NDI (INDI) improves the system robustness by decreasing the control law’s dependence on UAV’s model. Nevertheless, one of the most difficulties associated with INDI in order to have the best tracking performance is the controller gain selection. In this work, an Incremental NDI control strategy for fixed wing UAV is presented. The optimal set of controller gains is expressed as an optimization problem; the controller gains are subsequently determined through Incremental NDI using the Particle Swarm Optimization technique. This approach has a number of benefits, including a very effective global search algorithm, a straightforward implementation, and minimal algorithm parameters. The optimized controller is assessed through different simulation scenarios which include ideal case, with 10% uncertainty and with 30% uncertainty. The simulation affirms the PSO-INDI controller's superiority, effectiveness, and reliability. It demonstrates a 25% improvement in rise time, a 52% reduction in overshoot, and consistently low steady-state error even with increased model uncertainty, unmodeled dynamics and wind disturbances.