Scholarlink

<p>Immersive virtual reality (VR) navigation provides a naturalistic setting to study how humans integrate motivation, feedback, and social cues to guide exploration. Predicting navigation outcomes in such tasks is challenging because behavioral trajectories capture strong task-relevant dynamics but may be ambiguous in early decision stages, while EEG reflects internal cognitive processes yet is noisy and varies across individuals. To address this, we propose <Emphasis FontCategory="NonProportional">VRNavigationNet</Emphasis>, a unified multimodal prediction framework that jointly models EEG time–frequency representations, behavioral trajectories, and optional tabular subject attributes. Concretely, the model predicts trial-level navigation performance classes—defined by median-split accuracy in spatial landmark-finding and social-feedback-guided exploration tasks—from multimodal time-series recorded during each trial. The model follows an encode–fuse–predict design: modality-specific encoders convert each input stream into tokens, which are then integrated by a cross-modal Transformer to learn interaction-aware multimodal representations, enabling flexible inference when some modalities are missing. Experiments on the EEG Dataset of Exploring Navigation in Virtual Reality&#xa0;[<CitationRef CitationID="CR1">1</CitationRef>], comprising 60 participants performing spatial and social navigation under reward/punishment feedback, demonstrate that trajectories provide the strongest single-modality baseline, EEG offers complementary gains, and incorporating subject attributes further improves overall performance. Under a rigorous subject-wise evaluation protocol, VRNavigationNet achieves 80.1% average accuracy, significantly outperforming representative baselines including LSTM, single-modality Transformer, concatenation fusion, and late fusion (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(p&lt;0.01\)</EquationSource> </InlineEquation>). Comprehensive ablations validate the contribution of cross-modal fusion and key architectural components, highlighting the importance of modeling multimodal interactions for robust VR navigation prediction. The proposed framework provides a practical foundation for adaptive XR systems that leverage both embodied behavior and neural dynamics under heterogeneous data availability.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Cross-modal transformer fusion for EEG-trajectory navigation prediction in virtual reality

  • 期刊论文

  • 发表日期：

    2026年7月22日

查看全文
  • Shengrui Liu,
  • Yuting Pei,
  • Lei Ren

摘要

Immersive virtual reality (VR) navigation provides a naturalistic setting to study how humans integrate motivation, feedback, and social cues to guide exploration. Predicting navigation outcomes in such tasks is challenging because behavioral trajectories capture strong task-relevant dynamics but may be ambiguous in early decision stages, while EEG reflects internal cognitive processes yet is noisy and varies across individuals. To address this, we propose VRNavigationNet, a unified multimodal prediction framework that jointly models EEG time–frequency representations, behavioral trajectories, and optional tabular subject attributes. Concretely, the model predicts trial-level navigation performance classes—defined by median-split accuracy in spatial landmark-finding and social-feedback-guided exploration tasks—from multimodal time-series recorded during each trial. The model follows an encode–fuse–predict design: modality-specific encoders convert each input stream into tokens, which are then integrated by a cross-modal Transformer to learn interaction-aware multimodal representations, enabling flexible inference when some modalities are missing. Experiments on the EEG Dataset of Exploring Navigation in Virtual Reality [1], comprising 60 participants performing spatial and social navigation under reward/punishment feedback, demonstrate that trajectories provide the strongest single-modality baseline, EEG offers complementary gains, and incorporating subject attributes further improves overall performance. Under a rigorous subject-wise evaluation protocol, VRNavigationNet achieves 80.1% average accuracy, significantly outperforming representative baselines including LSTM, single-modality Transformer, concatenation fusion, and late fusion ( \(p<0.01\) ). Comprehensive ablations validate the contribution of cross-modal fusion and key architectural components, highlighting the importance of modeling multimodal interactions for robust VR navigation prediction. The proposed framework provides a practical foundation for adaptive XR systems that leverage both embodied behavior and neural dynamics under heterogeneous data availability.