<p>Road accidents caused by driver fatigue and cognitive overload remain a significant public safety concern. According to recent traffic safety data, drowsy driving contributes to thousands of fatal accidents each year, emphasizing the urgent need for intelligent driver monitoring systems. To address this, we propose an adaptive multimodal deep learning framework (AML) for real-time cognitive workload assessment and fatigue detection, leveraging the CL-Drive dataset: a multimodal repository of EEG (cognitive load), ECG (cardiac activity), EDA (electrodermal arousal), and gaze tracking (visual attention) captured from 21 participants during simulated driving across nine scenarios of escalating complexity. Our framework integrates a hybrid CNN–BiLSTM architecture to extract spatiotemporal features from raw physiological signals and gaze sequences, capturing localized spatial patterns and long-term temporal dynamics. These features are fused using a transformer-based network with cross-modal attention, which models interactions between modalities (e.g., correlating gaze fixation losses with EEG theta-band surges during distraction) and yields a 3.6 percentage-point absolute accuracy improvement over the strongest conventional fusion baseline under identical evaluation. To address individual variability and privacy, we combine personalized meta-learning—adapting to new drivers with as few as five windowed samples (<InlineEquation ID="IEq1"><EquationSource Format="TEX">\(\sim\)</EquationSource></InlineEquation>10 s of synchronized multimodal data) via episodic fine-tuning—with federated optimization, enabling decentralized model updates and reducing per-client data transfer by 38% through adaptive gradient compression. Experiments on CL-Drive demonstrate state-of-the-art performance under strictly cross-subject evaluation. Under subject-independent 5-fold cross-validation, AML achieves <InlineEquation ID="IEq2"><EquationSource Format="TEX">\(80.5 \pm 1.8\%\)</EquationSource></InlineEquation> accuracy on binary cognitive load classification without personalization, rising to <InlineEquation ID="IEq3"><EquationSource Format="TEX">\(91.8 \pm 1.2\%\)</EquationSource></InlineEquation> with <InlineEquation ID="IEq4"><EquationSource Format="TEX">\(K = 20\)</EquationSource></InlineEquation> calibration samples (<InlineEquation ID="IEq5"><EquationSource Format="TEX">\(\sim\)</EquationSource></InlineEquation>40 s). Under the more rigorous leave-one-subject-out (LOSO) protocol, AML reaches <InlineEquation ID="IEq6"><EquationSource Format="TEX">\(77.8 \pm 2.6\%\)</EquationSource></InlineEquation> without personalization and <InlineEquation ID="IEq7"><EquationSource Format="TEX">\(84.0 \pm 1.8\%\)</EquationSource></InlineEquation> with <InlineEquation ID="IEq8"><EquationSource Format="TEX">\(K = 20\)</EquationSource></InlineEquation> personalization, an improvement of 1.6 percentage points over the strongest published LOSO baseline on this dataset with a further 6.2–11.3 percentage points gained from personalization alone across the LOSO and 5-fold protocols. The framework exhibits robustness to real-world sensor noise (e.g., EEG/EDA motion artifacts) and achieves <InlineEquation ID="IEq9"><EquationSource Format="TEX">\(81.5 \pm 2.3\%\)</EquationSource></InlineEquation> LOSO accuracy with only <InlineEquation ID="IEq10"><EquationSource Format="TEX">\(K = 5\)</EquationSource></InlineEquation> samples (<InlineEquation ID="IEq11"><EquationSource Format="TEX">\(\sim\)</EquationSource></InlineEquation>10 s of calibration per new driver), critical for scalable in-vehicle deployment. By enabling privacy-aware, real-time monitoring of driver states, this work advances intelligent vehicle safety systems and provides a blueprint for adaptive multimodal learning in human-centric AI applications.</p>

Adaptive multimodal learning for driver cognitive state monitoring using transformer-based fusion with personalized meta-learning and federated optimization

期刊论文
出版模式：

开放获取
发表日期：

2026年5月6日

查看全文

G. Abinaya,
K. Dinakaran

摘要

Road accidents caused by driver fatigue and cognitive overload remain a significant public safety concern. According to recent traffic safety data, drowsy driving contributes to thousands of fatal accidents each year, emphasizing the urgent need for intelligent driver monitoring systems. To address this, we propose an adaptive multimodal deep learning framework (AML) for real-time cognitive workload assessment and fatigue detection, leveraging the CL-Drive dataset: a multimodal repository of EEG (cognitive load), ECG (cardiac activity), EDA (electrodermal arousal), and gaze tracking (visual attention) captured from 21 participants during simulated driving across nine scenarios of escalating complexity. Our framework integrates a hybrid CNN–BiLSTM architecture to extract spatiotemporal features from raw physiological signals and gaze sequences, capturing localized spatial patterns and long-term temporal dynamics. These features are fused using a transformer-based network with cross-modal attention, which models interactions between modalities (e.g., correlating gaze fixation losses with EEG theta-band surges during distraction) and yields a 3.6 percentage-point absolute accuracy improvement over the strongest conventional fusion baseline under identical evaluation. To address individual variability and privacy, we combine personalized meta-learning—adapting to new drivers with as few as five windowed samples (\(\sim\)10 s of synchronized multimodal data) via episodic fine-tuning—with federated optimization, enabling decentralized model updates and reducing per-client data transfer by 38% through adaptive gradient compression. Experiments on CL-Drive demonstrate state-of-the-art performance under strictly cross-subject evaluation. Under subject-independent 5-fold cross-validation, AML achieves \(80.5 \pm 1.8\%\) accuracy on binary cognitive load classification without personalization, rising to \(91.8 \pm 1.2\%\) with \(K = 20\) calibration samples (\(\sim\)40 s). Under the more rigorous leave-one-subject-out (LOSO) protocol, AML reaches \(77.8 \pm 2.6\%\) without personalization and \(84.0 \pm 1.8\%\) with \(K = 20\) personalization, an improvement of 1.6 percentage points over the strongest published LOSO baseline on this dataset with a further 6.2–11.3 percentage points gained from personalization alone across the LOSO and 5-fold protocols. The framework exhibits robustness to real-world sensor noise (e.g., EEG/EDA motion artifacts) and achieves \(81.5 \pm 2.3\%\) LOSO accuracy with only \(K = 5\) samples (\(\sim\)10 s of calibration per new driver), critical for scalable in-vehicle deployment. By enabling privacy-aware, real-time monitoring of driver states, this work advances intelligent vehicle safety systems and provides a blueprint for adaptive multimodal learning in human-centric AI applications.