With the advancement of deep learning technology, the accuracy of object detection in remote sensing images is significantly influenced by the data sources. Fusing optical and Synthetic Aperture Radar (SAR) images can effectively improve object detection accuracy by leveraging their individual strengths. However, challenges still exist in fusing optical and SAR images for object detection, particularly in effectively capturing object features and maintaining robustness in cloud-covered areas, which can adversely affect detection performance. To address these challenges, we propose the Instance and Cloud Masks Guided Multi-source Fusion Network (ICMGNet) for object detection, which integrates a SAM branch and a cloud branch. The SAM branch uses the segment anything model (SAM) to guide feature extraction and fusion using segmentation instance mask, focusing on object feature. Meanwhile, the cloud branch utilizes a pretrained cloud segmentation network to obtain cloud mask from optical images, guiding the fusion to enhance the weight of SAR feature in cloud-obscured regions. Our experimental results show that ICMGNet achieves higher object detection accuracy on OPTSAR and QXS-PART datasets compared to other state-of-the-art fusion object detection methods and single-source object detection methods.