In recent years, the widespread use of drones has brought convenience to our lives. At the same time, drones bring a certain degree of security threat in both civil and military aspects, so anti-drone technology has received more attention. Aiming at the problem of insufficient tracking rate during the tracking of UAVs, this paper proposes a Faster-SiamBAN method. The method is based on the SiamBAN network and modifies the architecture of the ResNet-50 network by first replacing the high-level outputs with the low-level outputs to reduce the offline training time of the algorithm. Then the symmetric downsampling structure is designed to solve the problem that the information between channel groups cannot be interacted. Meanwhile, the feature enhancement module and prediction module, which can utilize two different attention mechanisms to generate higher quality feature encoding, are added to improve the tracking success rate of the tracker. The experimental results show that the improved SiamBAN model reduces the single-frame tracking time by 0.007 s with the tracking success rate basically unchanged, which effectively shortens the UAV tracking time.