The automation of unmanned aerial vehicles (UAVs) has been driven in large part by vision object tracking methods with onboard cameras. However, random and complex real-world noise in captured imagery severely degrades the performance of state-of-the-art (SOTA) UAV trackers, especially under low-illumination conditions. To address this challenge, we propose a prompt-guided Taylorformer and design a plug-and-play, single-layer denoising network (SDT) aimed at suppressing heterogeneous noise and thereby improving UAV tracking performance. Specifically, our lightweight single-layer architecture employs minimal network depth to reduce computational overhead. We introduce prompt-guided Taylor self-attention (PTSA) and prompt-guided Taylor cross-attention (PTCA) to form a raw feature extraction (RFE) encoder and a multi-feature fusion (MFF) decoder, respectively, enhancing both feature extraction and fusion capabilities. In addition, we develop a multi-scale feed-forward network (MSFN) that more effectively leverages noise information across multiple receptive fields to further optimize network performance. Extensive experiments demonstrate that our proposed SDT achieves significantly denoising efficacy and substantially enhances UAV nighttime object tracking accuracy.