The COVID-19 crisis has hastened the uptake of educational classes on online platforms such as Zoom. To ensure the success of the educational process, instructors should monitor their students’ engagement levels during sessions. By assessing learners’ emotional and behavioral states, educators can refine their teaching strategies to improve the learning experience. However, accurately observing emotions and behaviors in a virtual environment is challenging. Therefore, there is a demand to automate engagement detection and prediction. Developing automatic detection systems to predict student engagement is crucial. This paper proposes a robust video-based engagement detection model that utilizes temporal feature extraction through the optical flow technique, combined with a pre-trained model. We experimented with various pre-trained Convolutional Neural Network (CNN) models, including MobileNetV2, ResNet101, EfficientNet, DenseNet161, VGG19, and InceptionV3. These models were fine-tuned using two public video datasets, DAiSEE and EmotiW23-EN, which are designed to evaluate student engagement in real-life online learning scenarios. Our research findings indicate that integrating optical flow with DenseNet161 significantly enhances engagement detection accuracy.