<p>Accurate monitoring of track irregularities is very helpful to improving the vehicle operation quality and to formulating appropriate track maintenance strategies. Existing methods have the problem that they rely on complex signal processing algorithms and lack multi-source data analysis. Driven by multi-source measurement data, including the axle box, the bogie frame and the carbody accelerations, this paper proposes a track irregularities monitoring network (TIMNet) based on deep learning methods. TIMNet uses the feature extraction capability of convolutional neural networks and the sequence mapping capability of the long short-term memory model to explore the mapping relationship between vehicle accelerations and track irregularities. The particle swarm optimization algorithm is used to optimize the network parameters, so that both the vertical and lateral track irregularities can be accurately identified in the time and spatial domains. The effectiveness and superiority of the proposed TIMNet is analyzed under different simulation conditions using a vehicle dynamics model. Field tests are conducted to prove the availability of the proposed TIMNet in quantitatively monitoring vertical and lateral track irregularities. Furthermore, comparative tests show that the TIMNet has a better fitting degree and timeliness in monitoring track irregularities (vertical <i>R</i>² of 0.91, lateral <i>R</i>² of 0.84 and time cost of 10&#xa0;ms), compared to other classical regression. The test also proves that the TIMNet has a better anti-interference ability than other regression models.</p>