Significant progress has been achieved in the fields of text, audio and image generation by utilizing brain signals for the reconstruction of visual stimulus. However, the development of 3D reconstruction from EEG signals faces significant challenges and remains in an early exploratory stage. To bridge this gap, we propose EEG2Mesh, a novel framework capable of efficiently converting EEG signals into high-quality 3D meshes. Considering the complexity and instability of EEG signals, the proposed EEG encoder enhances the representational capacity of EEG by extracting both time-domain and frequency-domain features. The 2D image generation module in our framework adopts a two-stage generation strategy to reconstruct high-quality 2D images via EEG embeddings, which provides accurate conditions for 3D reconstruction. Finally, through a multi-view diffusion model and sparse view reconstruction techniques, EEG2Mesh not only addresses the issue of floaters and cloud-like artifacts in the reconstructed results caused by complex background information but also better resolves multi-view consistency problems. Our work is pioneering in 3D reconstruction from EEG signals, indicating a promising new direction for research in neural decoding and visual reconstruction.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EEG2Mesh: High-Quality 3D Mesh Generation from EEG

  • Ying Fu,
  • Qiaoyu Chen,
  • Chenggang Song,
  • Taorui Li,
  • Zhipeng Yang,
  • Dongrui Gao

摘要

Significant progress has been achieved in the fields of text, audio and image generation by utilizing brain signals for the reconstruction of visual stimulus. However, the development of 3D reconstruction from EEG signals faces significant challenges and remains in an early exploratory stage. To bridge this gap, we propose EEG2Mesh, a novel framework capable of efficiently converting EEG signals into high-quality 3D meshes. Considering the complexity and instability of EEG signals, the proposed EEG encoder enhances the representational capacity of EEG by extracting both time-domain and frequency-domain features. The 2D image generation module in our framework adopts a two-stage generation strategy to reconstruct high-quality 2D images via EEG embeddings, which provides accurate conditions for 3D reconstruction. Finally, through a multi-view diffusion model and sparse view reconstruction techniques, EEG2Mesh not only addresses the issue of floaters and cloud-like artifacts in the reconstructed results caused by complex background information but also better resolves multi-view consistency problems. Our work is pioneering in 3D reconstruction from EEG signals, indicating a promising new direction for research in neural decoding and visual reconstruction.