Advanced cheating strategies are difficult to detect using traditional proctoring approaches, which frequently rely on single-mode inputs and simple feature extraction. The study introduces a sophisticated Online Exam Proctoring (OEP) system that uses a Hybrid Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) model to improve cheating detection in online learning. The suggested system incorporates multimodal data sources, such as audio signals, screen activity, video feeds, and application interactions, to overcome these constraints and greatly increase detection accuracy and dependability. Two significant advancements include an enhanced Davies Bouldin Score-based K-Means (DBS-KM) clustering technique for accurate object segmentation and the Kendall Rank Correlated Diamond Search (KRCDS) motion estimation algorithm, which improves temporal coherence in video sequences. Furthermore, high-dimensional data is reduced via the Schaeffer Weighted Kookaburra Optimization (SWKO) technique by removing the most separating characteristics from the multimodal input space, particularly when used in conjunction with LSTM layers for sequence analysis. The proposed method achieves a detection accuracy of 98.2%, surpassing conventional models such as standard DCNNs (95%) and RNNs (93.2%), according to thorough experimental evaluations on a customized dataset. Additionally, the system achieves a 37% reduction in false positives, improving practicality and dependability in real-world applications. With a completion time of just 56.8 s, the entire training process guarantees computational efficiency. According to these results, the proposed Hybrid CNN-LSTM OEP framework offers a scalable and effective way to raise the standard of online assessments.