Railroad tunnels, as important infrastructures where the line crosses complex mountains, require frequent and focused monitoring, especially at the tunnel entrance facilities connecting the mountains and slopes. Unmanned aerial vehicle (UAV)-based railroad tunnel entrance inspection has great potential to be an effective solution due to its highly maneuverable and wide aerial views. Unfortunately, current convolution neural network (CNN)-based methods struggle to accurately segment the railroad tunnel entrance facilities. This study presents a coarse-to-fine post-processing method that integrates the image processing algorithm and spatial topological representations between different facilities. First, outlier removal based on clustering is designed for coarse processing the segmentation results. Second, segmentation consistency verification and boundary refinement guided by structural adjacency relations progressively execute for fine optimizing and achieving the results. Finally, experimental results conducted on complex UAV railroad tunnel entrance dataset demonstrate that the proposed method exhibits strong generalization capability across different types of tunnels entrance and achieves stable and accurate structural segmentation.