This paper proposes an intelligent detection method of student classroom behavior based on an improved YOLOV8 model. First, a dataset comprising five categories of classroom misbehaviors, i.e., “using mobile phones”, “sleeping on the desk”, “looking down”, “whispering” and “looking around” is constructed. Second, an improved YOLOv8 model is designed to detect the student classroom behavior. To improve the model’s robustness in complex scenes, a spatial adaptive feature modulation network is introduced, leveraging multi-scale feature fusion. Then, the proposed method is validated on the constructed dataset, and the results demonstrate that the proposed method has improvement in recognition accuracy, i.e., increasing the mean average precision (mAP@0.5) to 81.7%. Furthermore, a visual system interface is developed using PyQt5. Leveraging integrated real-time video analysis and behavioral metrics, this work facilitates optimized teaching management and contributes to the development of a smart classroom ecosystem.