Scholarlink

<p>An inherent incongruity between the literal interpretation and the intended connotation characterizes sarcasm. Though several studies have targeted text-based unimodal sarcasm detection, multimodal sarcasm detection is still an evolving discipline. Emerging forms of multimodal data, such as memes, demonstrate that studying textual data alone may be insufficient for sarcasm detection. Additional contextual information conveyed by images can completely alter the perceived connotation of the text, thus indicating a sarcastic sentiment only when the text and image are studied in combination. This study presents a novel framework for multimodal sarcasm detection that can process input triplets, i.e., the input text and its associated image, as provided in the datasets, and a supplementary modality&#xa0;introduced in the form of image captions. The visual semantic representation provided by image captions offers an additional opportunity to capture the textual and visual content discrepancies. The key components of the proposed model are: (1) a textual feature extraction branch that utilizes a cross-lingual language model; (2) a visual feature extraction branch that incorporates a self-regulated residual ConvNet integrated with a lightweight spatially aware attention module; (3) image captions generated using an encoder-decoder architecture capable of reading text embedded in images; (4) distinct attention modules to effectively identify the incongruities between the text and two levels of image representations; (5)&#xa0;multi-level cross-domain semantic incongruity representation achieved through feature fusion. Compared with cutting-edge baselines, the proposed model achieves the best accuracy of 92.89% and 64.48%, respectively, on the Twitter multimodal sarcasm and MultiBully datasets.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Extracting cross-modal semantic incongruity with attention for multimodal sarcasm detection

  • 期刊论文

  • 发表日期：

    2025年7月17日

查看全文
  • Sajal Aggarwal,
  • Ananya Pandey,
  • Dinesh Kumar Vishwakarma

摘要

An inherent incongruity between the literal interpretation and the intended connotation characterizes sarcasm. Though several studies have targeted text-based unimodal sarcasm detection, multimodal sarcasm detection is still an evolving discipline. Emerging forms of multimodal data, such as memes, demonstrate that studying textual data alone may be insufficient for sarcasm detection. Additional contextual information conveyed by images can completely alter the perceived connotation of the text, thus indicating a sarcastic sentiment only when the text and image are studied in combination. This study presents a novel framework for multimodal sarcasm detection that can process input triplets, i.e., the input text and its associated image, as provided in the datasets, and a supplementary modality introduced in the form of image captions. The visual semantic representation provided by image captions offers an additional opportunity to capture the textual and visual content discrepancies. The key components of the proposed model are: (1) a textual feature extraction branch that utilizes a cross-lingual language model; (2) a visual feature extraction branch that incorporates a self-regulated residual ConvNet integrated with a lightweight spatially aware attention module; (3) image captions generated using an encoder-decoder architecture capable of reading text embedded in images; (4) distinct attention modules to effectively identify the incongruities between the text and two levels of image representations; (5) multi-level cross-domain semantic incongruity representation achieved through feature fusion. Compared with cutting-edge baselines, the proposed model achieves the best accuracy of 92.89% and 64.48%, respectively, on the Twitter multimodal sarcasm and MultiBully datasets.