In this paper, we propose an edge-supervised interactive fusion network (ESIF-Net) for curtain wall frames segmentation, which utilizes deep learning techniques to adapt to the needs of curtain wall installation localization in complex building environments. Specifically, we concatenate the RGB images and the corresponding depth images in the channel dimension into the backbone network to achieve multimodal feature fusion, and introduce the Edge Feature Extraction Module (EFE) to extract edge features. Then, the Edge Features Guide Interaction Module (EGI) is designed to interact and fuse the refined features with the edge features to enhance edge contour information in the top layer. Finally, we conduct comparative experiments on the curtain wall frame dataset, and demonstrate that the proposed model has better performance. The precision, recall, accuracy, and F1-score reach 85.62%, 83.42%, 96.31%, and 84.51%, respectively.