In the semantic segmentation of high-resolution remote sensing images, utilizing the normalized Digital Surface Model (nDSM) that provides height information as auxiliary data and fusing it with the visible image can improve the accuracy of segmentation. However, the better utilization of complementarity between different modal features has not been fully explored. In this work, we propose a new dual-branch and multi-stage Bimodal Fusion Rectification Network (BFRNet), which is end-to-end trainable. It consists of three modules: Channel and Spatial Fusion Rectification (CSFR) module, Edge Fusion Refinement (EFR) module, and Multiscale Feature Fusion (MSFF) module. The CSFR module integrates and rectifies multimodal features in both channel and spatial dimensions, achieving sufficient interaction and fusion between multimodal features. The EFR module obtains better multiscale edge features than single modality through feature fusion based on bimodal interactive edge attention and spatial gate, which helps to alleviate the edge loss of ground objects in single modality. The MSFF module is used to upsample and fuse multiscale features from EFR and CSFR to generate the final semantic segmentation results. The experimental results on the two public datasets, Vaihingen and Potsdam, provided by ISPRS, showcase the comparative advantage of the proposed method over other research methods.