Scholarlink

<p>String matching across heterogeneous modalities like text, images, audio, and video is critical for many applications, but traditional methods face challenges in modeling semantics, handling multimodal inputs, and providing explainable predictions. This paper presents a novel interpretable neural architecture aimed at facilitating multimodal fuzzy string matching via joint embeddings and cross-modal connections. The proposed approach constructs heterogeneous graphs encoding multimodal data from various sources like character sequences, visual regions, audio spectrogram slices, etc. These graph representations are jointly learned using graph attention and convolutional modules. For multimodal similarity scoring, we employ dual-encoder Siamese networks over the fused multimodal embeddings. Model initialization via meta-learning enables rapid adaptation, achieving superior performance with fewer examples compared to baselines across multiple domains and modalities. Technical details on multimodal data handling, model structure, optimization, and rigorous evaluations are provided. A unique strength is the incorporation of integrated gradients and cross-modal attention visualizations to offer post-hoc explainability of predictions, enhancing transparency and diagnosability. Extensive experiments on multimodal datasets demonstrate the effectiveness of joint multimodal encoding, external knowledge integration, few-shot cross-modal transfer ability, and interpretation capabilities—presenting a comprehensive multifaceted solution for real-world multimodal string matching applications. The paper introduces a new method of multimodal fuzzy string matching within the joint graph model of representation of text, image, audio, and video data. The approach uses graph neural networks to encode cross-modally and meta-learn few-shot adaptation. We present cross-modal attention and integrated gradients, which offer understandable and explainable predictions. The results of comprehensive experiments prove that the suggested model is an effective approach to various multimodal sets of data, and it is better than conventional methods regarding their precision and versatility.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Heterogeneous GNN-driven and meta-learned architecture for transparent multimodal string similarity estimation

  • 期刊论文

  • 发表日期：

    2026年1月17日

查看全文
  • Shaik Asha,
  • Sajja Tulasi Krishna

摘要

String matching across heterogeneous modalities like text, images, audio, and video is critical for many applications, but traditional methods face challenges in modeling semantics, handling multimodal inputs, and providing explainable predictions. This paper presents a novel interpretable neural architecture aimed at facilitating multimodal fuzzy string matching via joint embeddings and cross-modal connections. The proposed approach constructs heterogeneous graphs encoding multimodal data from various sources like character sequences, visual regions, audio spectrogram slices, etc. These graph representations are jointly learned using graph attention and convolutional modules. For multimodal similarity scoring, we employ dual-encoder Siamese networks over the fused multimodal embeddings. Model initialization via meta-learning enables rapid adaptation, achieving superior performance with fewer examples compared to baselines across multiple domains and modalities. Technical details on multimodal data handling, model structure, optimization, and rigorous evaluations are provided. A unique strength is the incorporation of integrated gradients and cross-modal attention visualizations to offer post-hoc explainability of predictions, enhancing transparency and diagnosability. Extensive experiments on multimodal datasets demonstrate the effectiveness of joint multimodal encoding, external knowledge integration, few-shot cross-modal transfer ability, and interpretation capabilities—presenting a comprehensive multifaceted solution for real-world multimodal string matching applications. The paper introduces a new method of multimodal fuzzy string matching within the joint graph model of representation of text, image, audio, and video data. The approach uses graph neural networks to encode cross-modally and meta-learn few-shot adaptation. We present cross-modal attention and integrated gradients, which offer understandable and explainable predictions. The results of comprehensive experiments prove that the suggested model is an effective approach to various multimodal sets of data, and it is better than conventional methods regarding their precision and versatility.