The pantograph and rigid overhead conductor system (ROCs) is an essential method for providing power to trains in tunnels. However, the complex wave behavior in ROC has become a critical factor limiting the speed increase of electric locomotives. In this paper, the filed measurement test based on the real ROCs is designed to analyze the wave reflection on the rigid conductor. And a hybrid algorithm based on adaptive Variational Mode Decomposition (VMD) and single-channel blind source separation is proposed to solve the severe modal aliasing occurring on the waves. Based on the single-mode vibration signal, the reflection coefficients of ROCs are obtained. It is indicated that the reflection coefficients vary with the clamp stiffness, ranging from 0.63 to 0.84 for the rigid suspension stiffness of 93,000 N/m and from 0.30 to 0.43 for the rigid suspension stiffness of 58,430 N/m. The experimental results also reveal that the reflection coefficients in the ROCs are not continuously distributed, contradicting the assumption of uniform distribution in previous simulation models.