The paper takes the Zhenguo Temple Pagoda as the research object. By using multi-scale analysis methods, the typical damage and long-term performance degradation problems are addressed to reveal the failure mechanism. Based on on-site non-destructive testing and concrete damage plasticity (CDP) model, a refined finite element model was established to conduct pushover analysis. The results showed that under horizontal load, structural damage exhibited a four stage evolution, from micro cracks at the entrance to vertical through failure zones. Nonlinear dynamic analysis confirms that vertical splitting is the main failure mechanism, but oblique damage is more pronounced in the middle and lower layers (1–4 layers) than in the pushover analysis. Based on the manual limit analysis of cantilever beam theory, it is clarified that vertical splitting is the most prone mechanism. The research results have verified the mechanism of brittle failure caused by stress concentration at the hole, providing a theoretical basis and technical path for the correction, reinforcement, and seismic protection of similar ancient pagodas.