Scholarlink

<p>Accurately predicting user gaze behavior is critical for intelligent and responsive interaction in six-degrees-of-freedom (6DoF) virtual reality (VR), enabling latency-sensitive functions such as attention-aware interaction and proactive rendering. In such scenarios, gaze behavior is jointly determined by head translation, head rotation, and eye gaze direction, and is further modulated by scene-dependent visual saliency as well as user and scene context. Existing methods often model only a subset of these factors and fuse multimodal cues via simple concatenation or static projection, which fails to enforce cross-modal geometric consistency and semantic alignment, resulting in limited generalization under constrained training data. We propose GBP-LLM, a foundation-model-driven framework that leverages a frozen large language model (LLM) as a unified sequence backbone for geometric-semantic fusion. Specifically, a Triangle Encoder models the geometric relationship between head pose and eye gaze, producing geometry-aware representations that provide the basis for aligning saliency cues with head-eye signals in a shared spatial space. Built upon this geometric basis, a structured prompting module extracts saliency information together with system context to guide multimodal reasoning. Experiments on our self-collected dataset and the public FixationNet-dataset demonstrate that GBP-LLM consistently outperforms conventional deep learning baselines on 6DoF gaze behavior prediction, and ablation studies verify the contribution of geometric encoding and each prompt component.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Gbp-llm: gaze behavior prediction in 6DoF VR via large language models

  • 期刊论文

  • 发表日期：

    2026年7月11日

查看全文
  • Ding Ding,
  • Wenjie Zhang,
  • Zicheng Liu,
  • Chang Qi,
  • Zheyu Cao,
  • Jinghui Zhang

摘要

Accurately predicting user gaze behavior is critical for intelligent and responsive interaction in six-degrees-of-freedom (6DoF) virtual reality (VR), enabling latency-sensitive functions such as attention-aware interaction and proactive rendering. In such scenarios, gaze behavior is jointly determined by head translation, head rotation, and eye gaze direction, and is further modulated by scene-dependent visual saliency as well as user and scene context. Existing methods often model only a subset of these factors and fuse multimodal cues via simple concatenation or static projection, which fails to enforce cross-modal geometric consistency and semantic alignment, resulting in limited generalization under constrained training data. We propose GBP-LLM, a foundation-model-driven framework that leverages a frozen large language model (LLM) as a unified sequence backbone for geometric-semantic fusion. Specifically, a Triangle Encoder models the geometric relationship between head pose and eye gaze, producing geometry-aware representations that provide the basis for aligning saliency cues with head-eye signals in a shared spatial space. Built upon this geometric basis, a structured prompting module extracts saliency information together with system context to guide multimodal reasoning. Experiments on our self-collected dataset and the public FixationNet-dataset demonstrate that GBP-LLM consistently outperforms conventional deep learning baselines on 6DoF gaze behavior prediction, and ablation studies verify the contribution of geometric encoding and each prompt component.