Scholarlink

<p>Current sign language machine translation systems rely on recognizing hand movements, facial expressions, and body postures, and natural language processing, to convert signs into text. While recent approaches use Transformer architectures to model long-range dependencies via positional encoding, they lack accuracy in recognizing fine-grained, short-range temporal dependencies between gestures captured at high frame rates. Moreover, their quadratic attention complexity leads to inefficient training. To mitigate these issues, we introduce ADAT, an Adaptive Transformer architecture that combines convolutional feature extraction, log-sparse self-attention, and an adaptive gating mechanism to efficiently model both short- and long-range temporal dependencies in sign language sequences. We evaluate ADAT on three datasets: the benchmark RWTH-PHOENIX-Weather-2014 (PHOENIX14T), the ISL-CSLTR, and the newly introduced MedASL, a medical-domain American Sign Language corpus. In sign-to-gloss-to-text translation, ADAT outperforms the state-of-the-art baselines, improving BLEU-4 by at least 0.1% and reducing training time by an average of 21% across datasets. In sign-to-text translation, ADAT consistently surpasses transformer-based encoder-decoder baselines, achieving a minimum of 0.5% gains in BLEU-4 and an average training speedup of 21.8% across datasets. Compared to the encoder-only and decoder-only baselines in sign-to-text, ADAT is at least 0.7% more accurate, despite being up to 12.1% slower due to its dual-stream structure.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ADAT novel time-series-aware adaptive transformer architecture for sign language translation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月28日

查看全文
  • Nada Shahin,
  • Leila Ismail

摘要

Current sign language machine translation systems rely on recognizing hand movements, facial expressions, and body postures, and natural language processing, to convert signs into text. While recent approaches use Transformer architectures to model long-range dependencies via positional encoding, they lack accuracy in recognizing fine-grained, short-range temporal dependencies between gestures captured at high frame rates. Moreover, their quadratic attention complexity leads to inefficient training. To mitigate these issues, we introduce ADAT, an Adaptive Transformer architecture that combines convolutional feature extraction, log-sparse self-attention, and an adaptive gating mechanism to efficiently model both short- and long-range temporal dependencies in sign language sequences. We evaluate ADAT on three datasets: the benchmark RWTH-PHOENIX-Weather-2014 (PHOENIX14T), the ISL-CSLTR, and the newly introduced MedASL, a medical-domain American Sign Language corpus. In sign-to-gloss-to-text translation, ADAT outperforms the state-of-the-art baselines, improving BLEU-4 by at least 0.1% and reducing training time by an average of 21% across datasets. In sign-to-text translation, ADAT consistently surpasses transformer-based encoder-decoder baselines, achieving a minimum of 0.5% gains in BLEU-4 and an average training speedup of 21.8% across datasets. Compared to the encoder-only and decoder-only baselines in sign-to-text, ADAT is at least 0.7% more accurate, despite being up to 12.1% slower due to its dual-stream structure.