Remote monitoring and intelligent diagnosis of operating condition are two key aspects for achieving cloud intelligent diagnosis of beam pumping unit. However, current manual detection method restricts real-time diagnostic efficiency. This paper established the Shufflenet_V2 lightweight model based on transfer learning for the proposed torque-angle dynamometer card to achieve effective and rapid diagnosis of working conditions. Firstly, a dataset of the torque-angle dynamometer card was constructed. Secondly, the Shufflenet_V2 pre-training model based on transfer learning was constructed and trained using the above recognition dataset. Finally, the overall performance of the model and the recognition performance under different working conditions were evaluated using model parameters, ACC and MCC evaluation indexes and compared with mainstream models. The results showed that compared with other models, the Shufflenet_V2 model reduced model parameters and at the same time, by applying transfer learning, the model’s recognition ability was greatly improved. The overall performance based on ACC and MCC was higher than 95%. In addition, when evaluating unbalanced working conditions, ACC cannot accurately represent the identification performance of each working condition. According to the MCC evaluation index, except for traveling valve leak, the recognition performance of each category is above 90%.