The widespread deployment of unmanned aerial vehicles (UAVs) has significantly enhanced modern technological applications while simultaneously introducing security challenges in urban environments, thereby elevating the importance of anti-UAV technologies. To address the critical challenge of detecting weak and small infrared UAV targets, this paper proposes a lightweight temporal target segmentation and recognition method based on CMSCM-LSTM-Unet. The proposed framework integrates multi-scale semantic feature fusion with temporal modeling to achieve pixel-level segmentation, effectively distinguishing targets from complex backgrounds and interference. Experimental results demonstrate that our algorithm successfully detects extremely small infrared targets with dimensions as minimal as 4 pixels, while achieving a recognition rate of 97.12% on public datasets. It indicates superior UAV identification performance compared to existing approaches, highlighting the method’s practical applicability in security-sensitive scenarios.