<p>Verifying whether the tracking result drifts during long-term tracking is a critical challenge. It’s difficult to select the optimal threshold in traditional threshold-based object drift verification criteria, and existing threshold-free object drift verification networks perform poorly in complex scenarios. To address these issues, we propose an object drift verification network based on multi-scale feature fusion and dual-template, using static and dynamic templates for joint verification. During the feature extraction stage, a multi-scale feature fusion module is introduced to adapt to changes in the object’s scale. Additionally, A template update strategy is devised to obtain high-quality dynamic templates for effective object drift verification. This network doesn’t require manual threshold setting and can be used as a plug-and-play module combined with a short-term visual tracking algorithm and a global re-detection module for long-term tracking. The proposed network forms four long-term tracking algorithms by integrating with four short-term visual tracking algorithms (DiMP50, PrDiMP, TrDiMP, and TransT). Extensive experiments on datasets like LaSOT, UAV20L, VOT2018-LT, and VOT2020-LT demonstrate significant improvements in long-term tracking performance. On the UAV20L dataset, the success rate and precision improved by 8.5% and 9.4%, respectively, compared to the base algorithm TrDiMP. On the VOT2020-LT dataset, the F-score improved by 6.1% compared to the base algorithm PrDiMP. Moreover, the proposed network achieves a verification speed of 220 FPS, with minimal impact on long-term tracking speed.</p>