The recognition of small unmanned aerial vehicles (UAVs) has garnered increasing interest due to their potential threat to security. UAV recognition is a significant challenge due to their small radar cross section and potential masking in dynamic clutter environments. Dynamic Data-Driven Applicaiton Systems (DDDAS) can exploit deep neural networks (DDNs) to learn in specific environments, but the gap between pre-deployment training and post-deployment test data can significantly degrade automatic target recognition (ATR) performance. To bridge this gap, this paper proposes a semantic segmentation based physics-aware generative adversarial network (GAN) for more accurately synthesizing drone micro-Doppler ( \(\mu \) D) signatures across disparate clutter profiles. CAD-based models of drones are used to train a Semantic Segmentation (SS) DNN, which can discriminate target and clutter pixels. The SS-DNN is then incorporated into a GAN to better inform the discriminator of the target component of the data. The UAV \(\mu \) D signatures synthesized using the proposed Semantic Segmentation-based Physics-Aware GAN (SS-PhGAN) is then evaluated from the perspective of kinematic fidelity in high signal-to-interference-plus-noise-ratio (SINR) scenarios. The proposed approach is evaluated using real data collected from a quadcopter and hexacopter in an open field or near a tree to capture data across different SINR levels. Our results provide important insights on DNN design that can effectively operate in dynamic clutter environments.