Surface defect inspection of railways is critical to ensure safe transportation of railway system. However, existing challenges such as irregular defect shapes and similarity between foreground and background hinder surface defect inspection. Previous research methods usually focus on single-level cross-modal fusion, which may not fully utilize multi-modal information. To address this, we introduce a novel framework, termed HM3DNet, which benefits defect detection performance by exploring both hierarchical cross-modal features and depth-enhanced volumetric fusion features. Specifically, we propose a Cross-modal Feature Enhancement (CFE) module to enable the mutual enhancement and complementary fusion of features from the two modalities. To effectively capture rich multi-modal information, we design a Multi-modal Feature Integration (MFI) module to aggregate cross-modal features from CFE and modality-specific features from each decoder. In addition, we design a Depth-Enhanced Volumetric Fusion Network (DeV-FuseNet), which leverages inflated 3D encoder to perform depth-aware fusion of RGB and depth modalities. Moreover, we adopt the dual-stream modality-specific decoder network to capture hierarchical modality-specific information. Experimental results on the NEU RSDDS-AUG dataset revealed that HM3DNet outperformed state-of-the-art (SOTA) methods. We also validate HM3DNet’s generalizability on three other benchmark datasets, where experiment results show that it competes well.