Scholarlink

<p>There exist certain deficiencies, including the absence of dynamic weight allocation, inadequate fine-grained semantic alignment, and weak noise robustness. To tackle these problems, this paper presents the Multimodal Dynamic-aware Translation Model (MD-Trans) based on the Transformer architecture. By incorporating the Cross-modal Attention Enhancement (CMAE) with contrastive learning, a text-image-speech triple contrastive learning task is formulated using the InfoNCE loss function. Integrating Tri-directional Cross Attention with cultural knowledge graph embedding, it enforces the alignment of the latent semantic space distributions of heterogeneous modalities, resolves the cross-cultural mapping deviation problem of culture-loaded words, and enhances the cultural appropriateness of translation. A lightweight Dynamic Weight Allocation (DWA) module is devised to assess the modal quality confidence in real-time, dynamically adjust the modal weights, and suppress the noise propagation from low-quality inputs to the translation results. A hierarchical feature fusion is established. Based on entity detection and scene graph modeling, two-stream parallel processing is accomplished. Fine-grained entity-level features and coarse-grained scene-level semantics are respectively extracted, and dynamic gating weighted fusion is employed. While ensuring computational efficiency, the semantic coherence of long-sentence translation is augmented. Experiments demonstrate that on the multimodal test sets constructed from WMT2022, Flickr30k, and LibriSpeech, the BLEU-4 of MD-Trans reaches 48.9, representing an increase of 16.2% compared to the single-modal baseline. In the mixed-noise scenario, its Translation Error Rate (TER) is 23.1, signifying a decrease of 26.3% compared to the baseline. Through dynamic weight allocation, when the speech is noisy (SNR = 8dB), the speech weight automatically reduces to 0.18, significantly suppressing noise propagation. In the cross-scene translation task, the TER of MD-Trans for spoken dialogue is as low as 25.3, and the MC-Score for image description is as high as 75.4, outperforming mainstream models. MD-Trans can offer accurate translation support for scenarios such as healthcare, cross-border e-commerce, and educational collaboration.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Strategies for improving the accuracy of English translation through multimodal fusion deep learning

  • 期刊论文

  • 发表日期：

    2026年1月23日

查看全文
  • Lan SHEN,
  • Yang Bai,
  • Jing Duan

摘要

There exist certain deficiencies, including the absence of dynamic weight allocation, inadequate fine-grained semantic alignment, and weak noise robustness. To tackle these problems, this paper presents the Multimodal Dynamic-aware Translation Model (MD-Trans) based on the Transformer architecture. By incorporating the Cross-modal Attention Enhancement (CMAE) with contrastive learning, a text-image-speech triple contrastive learning task is formulated using the InfoNCE loss function. Integrating Tri-directional Cross Attention with cultural knowledge graph embedding, it enforces the alignment of the latent semantic space distributions of heterogeneous modalities, resolves the cross-cultural mapping deviation problem of culture-loaded words, and enhances the cultural appropriateness of translation. A lightweight Dynamic Weight Allocation (DWA) module is devised to assess the modal quality confidence in real-time, dynamically adjust the modal weights, and suppress the noise propagation from low-quality inputs to the translation results. A hierarchical feature fusion is established. Based on entity detection and scene graph modeling, two-stream parallel processing is accomplished. Fine-grained entity-level features and coarse-grained scene-level semantics are respectively extracted, and dynamic gating weighted fusion is employed. While ensuring computational efficiency, the semantic coherence of long-sentence translation is augmented. Experiments demonstrate that on the multimodal test sets constructed from WMT2022, Flickr30k, and LibriSpeech, the BLEU-4 of MD-Trans reaches 48.9, representing an increase of 16.2% compared to the single-modal baseline. In the mixed-noise scenario, its Translation Error Rate (TER) is 23.1, signifying a decrease of 26.3% compared to the baseline. Through dynamic weight allocation, when the speech is noisy (SNR = 8dB), the speech weight automatically reduces to 0.18, significantly suppressing noise propagation. In the cross-scene translation task, the TER of MD-Trans for spoken dialogue is as low as 25.3, and the MC-Score for image description is as high as 75.4, outperforming mainstream models. MD-Trans can offer accurate translation support for scenarios such as healthcare, cross-border e-commerce, and educational collaboration.