<p>The layout analysis of Chinese ancient mathematical books presents significant challenges. Elements for detection are densely distributed, and the morphological similarity between mathematical symbols and Chinese characters complicates layout analysis. Additionally, ink stain degradation and noise interference from historical preservation constrain traditional methods. Given current object detection algorithms’ limitations in handling high noise, dense detection, and similar category distinction, this paper proposes the AL-YOLO (Ancient Layout You Only Look Once) detection framework. Enhanced through a triple architecture, AL-YOLO improves complex layout division accuracy. Firstly, a hybrid degradation-aware convolution module (HDAC) was designed in the feature extraction process to enhance the model’s feature representation ability and alleviate the feature degradation problem in deep networks. Asymmetric Convolution Net (ACNet) is introduced during feature extraction to enhance geometric feature representation in noisy regions, combined with an Efficient Multi-Scale Attention (EMA) mechanism to dynamically focus on key texture features, improving detection of blurry characters and damaged symbols. Second, the traditional Path Aggregation Network (PAN) is improved into a Bidirectional Feature Pyramid Network (BiFPN) to enhance perception of symbols and dense text via cross-layer feature fusion. Finally, Weighted Intersection over Union (WIoU) is adopted as the bounding box regression loss function, correcting element location deviations. Experiments show that AL-YOLO can perform fine detection of 17 object categories, with an average precision (mAP0.5) increase of 3.5% compared to the SOTA model. This method demonstrates significant advantages in addressing noise interference, dense element distribution, and category distinction, providing an effective technical solution for mathematical book digital preservation.</p>