Scholarlink

<p>This paper aims to design and evaluate a text-speech-image multimodal collaborative English translation system. Addressing the issues of coarse-grained modal semantic alignment and the difficulty in balancing inference efficiency and translation quality in current text-speech-image multimodal English translation systems, a multimodal collaborative translation system based on fine-grained cross-modal alignment and an adaptive gating fusion mechanism is proposed. First, Vision Transformer (ViT)-Base, wav2vec 2.0, and multilingual BART with 50 languages (mBART-50) are used to extract context-aware features of image regions, speech frames, and text words, respectively. Fine-grained alignment of triples is achieved through contrast loss in the shared semantic space and a cross-modal mask reconstruction task. Then, a modality-aware gating network is designed to dynamically adjust the fusion weights of image and speech at the word level, guided by text, and a single-layer lightweight cross-attention module with fixed structural parameters is embedded to reduce decoding overhead. A two-stage training strategy is employed. First, end-to-end fine-tuning is used to jointly optimize the negative log-likelihood, contrastive, and reconstruction losses. Then, a policy gradient-based REINFORCE algorithm with specific sampling and variance reduction strategies is applied for optimization, jointly using Bilingual Evaluation Understudy (BLEU), Contrastive Language-Image Pretraining Score (CLIPScore), and latency-aware reward. Experiments on the 3AM and Flickr8k-En datasets demonstrate that the proposed method outperforms baseline models across multiple metrics, including text quality, cross-modal consistency, speech adaptability, and system efficiency. Specifically, BLEU-4 scores reach 42.7; CLIPScore is 0.802; Word Error Rate (WER) is 8.3%; end-to-end latency is 420&#xa0;ms. This method effectively improves the disambiguation capability, cross-modal consistency, and system real-time performance of multimodal translation, and preliminarily verifies the balanced advantage of the proposed architecture in multimodal semantic collaborative modeling and practical deployment scenarios.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Design and performance evaluation of text-speech-image multimodal collaborative English translation system

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月24日

查看全文
  • Kai Lu,
  • Yimin Jiang

摘要

This paper aims to design and evaluate a text-speech-image multimodal collaborative English translation system. Addressing the issues of coarse-grained modal semantic alignment and the difficulty in balancing inference efficiency and translation quality in current text-speech-image multimodal English translation systems, a multimodal collaborative translation system based on fine-grained cross-modal alignment and an adaptive gating fusion mechanism is proposed. First, Vision Transformer (ViT)-Base, wav2vec 2.0, and multilingual BART with 50 languages (mBART-50) are used to extract context-aware features of image regions, speech frames, and text words, respectively. Fine-grained alignment of triples is achieved through contrast loss in the shared semantic space and a cross-modal mask reconstruction task. Then, a modality-aware gating network is designed to dynamically adjust the fusion weights of image and speech at the word level, guided by text, and a single-layer lightweight cross-attention module with fixed structural parameters is embedded to reduce decoding overhead. A two-stage training strategy is employed. First, end-to-end fine-tuning is used to jointly optimize the negative log-likelihood, contrastive, and reconstruction losses. Then, a policy gradient-based REINFORCE algorithm with specific sampling and variance reduction strategies is applied for optimization, jointly using Bilingual Evaluation Understudy (BLEU), Contrastive Language-Image Pretraining Score (CLIPScore), and latency-aware reward. Experiments on the 3AM and Flickr8k-En datasets demonstrate that the proposed method outperforms baseline models across multiple metrics, including text quality, cross-modal consistency, speech adaptability, and system efficiency. Specifically, BLEU-4 scores reach 42.7; CLIPScore is 0.802; Word Error Rate (WER) is 8.3%; end-to-end latency is 420 ms. This method effectively improves the disambiguation capability, cross-modal consistency, and system real-time performance of multimodal translation, and preliminarily verifies the balanced advantage of the proposed architecture in multimodal semantic collaborative modeling and practical deployment scenarios.