This article aims to explore the application of deep learning (DL) technology in the analysis and optimization of track and field athletes’ movement technology. A DL model combining You Only Look Once version 5(YOLOv5), Improved Convolutional Neural Network (CNN) and Action Classifier is proposed to accurately identify and analyze the action characteristics of track and field athletes. Through well-designed experiments, the excellent performance of the model in track and field athletes’ action recognition is verified, and the relationship between recognition time and model complexity is discussed. The results show that YOLOv5 algorithm is superior to the traditional algorithm in accuracy and recall, especially in complex scenes and challenging conditions. In addition, the key role of action feature extraction in track and field athletes’ action recognition is found, and the law that increasing the model complexity appropriately can improve recognition efficiency is also found. The analysis and optimization method of track and field athletes’ movement technology based on DL proposed in the study provides a scientific evaluation method for track and field training. Future work will continue to study the application of DL technology in the field of track and field to contribute more to the development of track and field.