The main challenge in object tracking is how to track a target in real time and accurately in complex and changing environments. Correlation filters (CFs) have unique advantages in the field of target tracking and are widely used in unmanned aerial vehicle (UAV) platforms. We propose a new tracking algorithm that introduces adaptive background residuals to reduce the environmental interference during tracking. Experimental results on two benchmark datasets show that our tracking algorithm achieves first place in accuracy and success rate compared to nine other state-of-the-art tracking algorithms. Our tracker has excellent tracking performance.