Intelligent transportation systems rely heavily on robust scene understanding under varying environmental conditions for decision-making and driver assistance. In this paper, we introduce a novel MsResNet: Multi-scale Edge-Enhanced ResNet for RGB-thermal image segmentation, combining RGB and thermal images to address challenges introduced by varying lighting conditions in scene understanding. MsResNet incorporates multi-scale guided filtering to enhance edge definitions and contrast, and it uses attention-based cross-fusion to dynamically integrate features from RGB and thermal modalities across their spatial dimensions. In addition, a weighted compound loss function refines the predictions at the region, boundary, and pixel levels. Experimental results in the MF and KAIST datasets suggest that MsResNet performs on par with current state-of-the-art models, with reduced parameters and inference time, achieving IoU scores of 59.8 and 50.24, respectively. These results demonstrate the suitability of the model for real-time ADAS applications and scene understanding.