Real-Time Pilot Cognitive Monitoring via Multimodal Fusion of Gaze and Control Signals: A Wavelet Scalogram Deep Learning Approach
摘要
Aviation safety critically depends on continuously monitoring pilot cognitive states, yet conventional assessments, ranging from subjective questionnaires to offline performance tests, often fail to capture rapid, in-flight fluctuations in attention and workload. In this study, we present a novel real-time cognitive monitoring framework that leverages high-frequency eye-tracking data combined with pilot control inputs to continuously estimate the exceedance shape factor, an indicator of deviations from ideal flight parameters and attention lapses. We extract time–frequency features from raw uncalibrated gaze signals using the Continuous Wavelet Transform, generating scalograms that serve as inputs to deep Convolutional Neural Networks. By incorporating cross-correlation analysis between gaze and control signals, we add a valuable modality for assessing eye–hand coordination while maintaining a single-modal training paradigm. This design not only enriches the feature set through multimodal fusion but also ensures computational efficiency and reduced data requirements, thanks to the expressive power of wavelet-based scalograms. Experimental evaluation on a dataset of approximately 1,600 scalogram images collected over 45 sessions demonstrates that architectures such as Googlenet and InceptionResNetV2 achieve high predictive accuracy, with the best model reaching an average test R2 of ~ 0.75. These findings underscore the potential of our framework to enhance situational awareness monitoring and improve flight safety through adaptive, in-flight cognitive state estimation. Future work will focus on expanding dataset diversity, integrating additional sensor modalities, and validating the approach in operational environments.