In order to effectively improve students’ behavior management in the classroom and teaching effectiveness assessment and other issues, the automatic identification of students’ classroom behavior method using YOLOv8s model is proposed. Introducing Squeeze and Excitation (SE) attention mechanism in YOLOv8s to enhance the model’s learning ability Additionally, it replaced the original Spatial Pyramid Pooling-Fast (SPPF) layer with Simplified Similarity-aware Spatial Pyramid Pooling with Features (SimSPPF), the network has improved detection ability and velocity for a wide range of target sizes. Furthermore, the loss function in YOLOv8s was substituted with the Smoothed Intersection over Union (SIOU), increasing the model’s detection accuracy. The experimental findings indicate that the improved model achieves a 13.2% higher recognition accuracy compared to the original model, reaching 87.7%. This algorithm is effective in detecting multi-person classroom behaviors in an educational setting. It not only helps teachers to better manage the classroom order, but also strengthens teaching effectiveness evaluation, contributing to the intelligent advancement of the education sector.