Digital technologies like deep learning (DL) and image processing have been used to enhance the performance of manual structural damage inspections in heritage, making them time-saving, cost-effective, and accurate. However, obtaining defect images for the dataset is labor and time-intensive, especially for cultural heritage structures (CHS), due to the limited availability of public datasets. Additionally, the labeling process presents another challenge. Therefore, the insufficient number of images in the dataset is an obstacle to improving the accuracy of DL models for detecting damage in such structures. In this research work, instead of increasing the number of dataset images, the objective is to improve the accuracy of DL models for damage detection (DD) through transfer learning (TL). About 1800 CHS crack images, categorized into three classes (brick, stone, and earthen materials), are utilized. Various TL-based DL models (such as inductive and transductive TL approaches) were applied to classify the crack dataset and were compared to models without TL. Based on the analysis, combining TL approaches with a lightweight convolutional neural network named MobileNetV2 improved classification accuracy to 94.67%. Subsequently, transductive TL using the ImageNet dataset achieved 93.67% accuracy, and inductive TL reached 92.33% accuracy with testing data. In contrast, employing MobileNetV2 without TL plateaued at 33.33% accuracy on the test data. Moreover, the sequential model without TL achieved 81.67%. This highlights that TL approaches can enhance the performance of DL methods without increasing the total number of dataset images.