Heritage structures are invaluable cultural assets requiring careful preservation. Traditional inspection methods for defect detection are time-consuming, subjective, and often inaccessible. This paper compares the performance of selected machine learning techniques (support vector machines, decision trees, k-nearest neighbour, and convolutional neural networks) in detecting wall defects on heritage structures in South Africa. A dataset of images made up of crack, spall, and intact areas was collected from the Castle of Good Hope and Dal Josafat located in Cape Town, South Africa. Experimental results show CNN produced the highest accuracy, precision, and recall scores among all models. This research contributes to developing a cutting-edge, non-invasive, and cost-effective tool for heritage preservation, enabling proactive maintenance and ensuring the longevity of South African cultural heritage.