Assessing building damage through high-resolution remote sensing images is crucial for humanitarian relief. The commonly used Siamese Network is inadequate for fitting complex features in building damage assessment. To address the abovementioned issues, this paper proposes Siamese Kolmogorov-Arnold Networks (SiameseKAN). In order to better accommodate the complex high-level features of multiple damage levels, a KAN Block is introduced at the bottleneck layer of SiameseKAN near the encoder-decoder architecture. In addition, the model proposes a rectangular field loss and an object gap loss to focus on the damage level characteristics within the building. The experimental results indicate that this method surpasses other methods, resulting in an improvement of the total score on the xBD dataset by 4.8%.