Scholarlink

<p>Generating natural 3D human motion from given text is a crucial research hotspot in the field of computer vision in recent years. However, due to the complexity of human motion and the challenge of learning cross-modal relationships between text and motion, accurately generating corresponding continuous motion from complex text remains a formidable problem, which requires a deep understanding of linguistic nuances and human motion analysis. To address these challenges, we propose CEDT2M, a framework for text to motion generation based on cross-modal mixture of encoder-decoder. Specifically, our CEDT2M introduces a multitask learning approach to jointly train unsupervised motion-text alignment and motion generation without relying on labeled data. This approach enables the model to learn multiple related tasks simultaneously and capture the internal relationship between these tasks. In addition, our CEDT2M combines a visual transformer encoder for encoding skeleton-based motion sequences with a BERT-like text encoder, enabling efficient processing of both modalities. Extensive experiments demonstrate that CEDT2M can achieve competitive zero-shot retrieval performance on benchmark datasets like KIT and HumanML3D, while also outperforming the state-of-the-art results. The code will be available at: <a href="https://github.com/lplp926120/CEDT2M.">https://github.com/lplp926120/CEDT2M.</a></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

CEDT2M: text-driven human motion generation via cross-modal mixture of encoder-decoder

  • 期刊论文

  • 发表日期：

    2025年3月10日

查看全文
  • XiangYang Wang,
  • Peng Li,
  • Rui Wang

摘要

Generating natural 3D human motion from given text is a crucial research hotspot in the field of computer vision in recent years. However, due to the complexity of human motion and the challenge of learning cross-modal relationships between text and motion, accurately generating corresponding continuous motion from complex text remains a formidable problem, which requires a deep understanding of linguistic nuances and human motion analysis. To address these challenges, we propose CEDT2M, a framework for text to motion generation based on cross-modal mixture of encoder-decoder. Specifically, our CEDT2M introduces a multitask learning approach to jointly train unsupervised motion-text alignment and motion generation without relying on labeled data. This approach enables the model to learn multiple related tasks simultaneously and capture the internal relationship between these tasks. In addition, our CEDT2M combines a visual transformer encoder for encoding skeleton-based motion sequences with a BERT-like text encoder, enabling efficient processing of both modalities. Extensive experiments demonstrate that CEDT2M can achieve competitive zero-shot retrieval performance on benchmark datasets like KIT and HumanML3D, while also outperforming the state-of-the-art results. The code will be available at: https://github.com/lplp926120/CEDT2M.