Due to the labor-intensive nature of pixel-wise annotation for remote sensing images, which feature complex backgrounds, high resolution, and diverse target structures, current research focuses on exploring semantic segmentation networks that can be trained on existing data and adapted to cross-domain datasets. In this paper, we propose AMDFormer, combined with Unsupervised Domain Adaptation (UDA), which includes an innovative Adaptive Temperature for Sampling (ATS) rare classes to alleviate the class imbalance problem and dynamically regulate sampled data distributions, by initially increasing the sampling ratio of rare classes and later aligning with the dataset’s distribution. Meanwhile, we propose adjusting the criterion for cross-domain image mixing strategy with Modulated Dynamic Threshold (MDT), which leverages the principle that aggregated pixel quality is generally higher than individual pixel quality, driving the model to generate higher quality labels throughout the training procedure. Experimental results on the ISPRS datasets and LoveDA dataset demonstrate the effectiveness of our proposed model.