In the existing material damage identification technology in civil engineering, traditional identification mainly relies on manual inspection and simple instrument detection, which is time-consuming and laborious, and difficult to achieve large-scale and real-time damage evaluation. This article aims to establish a material damage identification and evaluation system based on Convolutional Neural Network (CNN), and apply it to civil engineering structures to achieve automated identification and classification of structural damage. This article intends to take civil engineering and architecture as the research object, and collect various types of damage pictures on site, including cracks, rust, peeling, etc. It adopts methods such as image preprocessing to improve image quality, and adopts data augmentation methods to increase sample size and enhance the model's generalization performance. This article adopts a deep convolutional neural network structural damage identification method that integrates multiple layers of convolutional layers, pooling layers, and fully connected layers to achieve automatic extraction and efficient identification of structural damage features. The convolutional neural network model can still ensure the recognition accuracy of the model well in the presence of background interference such as rain and shadows. Compared with the traditional manual detection method, the recognition accuracy of CNN model method is higher than 15%. The results of material damage identification and evaluation in this paper have certain reference significance for establishing a scientific engineering structure evaluation system and formulating new testing standards.