In the daily operation and maintenance of power poles, foreign object entanglement, unintended contact between power lines and foreign materials, and insulator defects are key contributors to tower malfunctions and operational anomalies. In order to solve the low accuracy caused by the diverse types and sizes of foreign objects on towers, as well as the computational complexity of existing detection models that fail to meet application requirements, a multi-scale adaptive dual-layer routing attention-based anomaly detection model for towers is proposed. Specifically, to account for the wide variety of foreign bodies and different sizes in the detection of foreign bodies in towers, the Bi-Level Routing Attention (BRA) mechanism was structurally optimized. Building on this, an Adaptive Bi-Level Multi-scale attention module (ABMD) was proposed. In the backbone part of the model, MobileNetv3 is introduced as the backbone network to replace the original backbone network adopted by the YOLOv5 framework, resulting in the final model named ABMnv3Net. Experimental results show that the ABMnv3Net network can efficiently detect abnormal parts of towers of different types and sizes, and the average accuracy is increased by 3.3% and the number of parameters is reduced by 36% compared with the baseline model, which has outstanding advantages in the accuracy and model complexity of tower anomaly detection.