Nighttime imagery is not only challenged by insufficient ambient light but also significantly impacted by intense artificial illumination such as glare and headlight halos. Most existing enhancement techniques concentrate on brightening dark areas, often at the cost of exacerbating overexposed regions. In this work, we introduce RefineFuse, a novel multi-scale fusion framework tailored for high-fidelity night vision by integrating infrared and visible spectrum imagery. The proposed system decouples the fusion of structural and semantic information: the former leverages an Exposure-Aware Fusion Module (EAFM) to mitigate uneven lighting, while the latter employs a Semantic Alignment Module with Detection Cues (SADC) to enrich target-related semantics via cross-task guidance from a detection backbone. Additionally, we introduce a Contrast-Aware Illumination Loss designed to regularize the illumination consistency of the fused output in an unsupervised manner. Extensive experiments confirm that RefineFuse not only enhances perceptual quality but also substantially improves object detection reliability under complex night lighting. The implementation will be publicly available upon completion of the review process.