Scholarlink

Sign Language Production (SLP) aims to produce sign sequences that are semantically consistent with spoken language. Currently, the process of Gloss to Pose does not sufficiently explore the local information of finger movement in SLP. This paper proposes a dual-channel mixed Token Progressing Unit model with Graph Convolution Network (GCN) and Transformer (TPU-GTLG) to address spatial-temporal local and global information. TPU-GTLG comprises a spare attention encoder and a mixed token progressing unit decoder. In the encoder, we design a spare attention mechanism with a Top-k selection. This mechanism allows the top k elements with the highest scores to all have a chance of being selected. This method reduces the interference of irrelevant information. In the decoder, a parallel dual-channel approach was devised, combining the Transformer and GCN, to integrate the global representation of the Transformer with the local information of the graph model. Attention fusion is then used to aggregate the features from the dual-channel and global features, generating semantically continuous natural sign language. Experimental results show that TPU-GTLG achieves superior performance on the PHOENIX14T and our collected dataset CVSLD.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Dual-Channel Mixed Token Progressing Unit Model for Sign Language Production with Global Transformer and Local GCN

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Yarun Yang,
  • Qingshan Wang,
  • Qi Wang,
  • Peng Liu

摘要

Sign Language Production (SLP) aims to produce sign sequences that are semantically consistent with spoken language. Currently, the process of Gloss to Pose does not sufficiently explore the local information of finger movement in SLP. This paper proposes a dual-channel mixed Token Progressing Unit model with Graph Convolution Network (GCN) and Transformer (TPU-GTLG) to address spatial-temporal local and global information. TPU-GTLG comprises a spare attention encoder and a mixed token progressing unit decoder. In the encoder, we design a spare attention mechanism with a Top-k selection. This mechanism allows the top k elements with the highest scores to all have a chance of being selected. This method reduces the interference of irrelevant information. In the decoder, a parallel dual-channel approach was devised, combining the Transformer and GCN, to integrate the global representation of the Transformer with the local information of the graph model. Attention fusion is then used to aggregate the features from the dual-channel and global features, generating semantically continuous natural sign language. Experimental results show that TPU-GTLG achieves superior performance on the PHOENIX14T and our collected dataset CVSLD.