The growing awareness of property safety inspections among governments and the public has fueled the demand for efficient, automated methods of damage assessment. Despite this, there is a notable scarcity of datasets specifically designed for house damage classification tasks. To address this gap, this paper presents the Build Damage Classification (BDC) Dataset, an enhanced dataset built upon xBD, incorporating three distinct sub-datasets for building damage classification. Additionally, to assess the impact of noise and low-quality data on model performance, two contrastive learning methods-DINOv2 and MoCo v2-are applied to classify property damage resulting from natural disasters. Experimental results reveal that DINOv2 significantly outperforms traditional CNNs and MoCo v2, with a notable improvement of approximately 20% in precision, recall, and F1 scores on the highly imbalanced and low-quality BDC dataset. Moreover, attention maps and gradient visualization techniques are used to explain the performance differences between the models.