Scholarlink

<p>Multimodal summarization methods often depend on supervised learning, which limits their generalization to unseen data. To address this, we propose Reinforcement Learning-based Multimodal Summarization (RLMS), a novel framework that generates multimodal output (MO) summaries text paired with contextually relevant images from multimodal input (MI) data containing both text and images. RLMS is structured into three layers: (1) a Multimodal Feature Extraction (MFE) layer that captures latent features from text and images, (2) a Reinforcement Learning (RL) layer that generates textual summaries using policy gradients, and (3) a Multimodal Summarization (MS) layer that integrates the generated text with the most relevant image to form the final summary. The RL layer introduces a summarization-based policy gradient (SPG) as a reward function to directly optimize summary quality, ensuring contextual accuracy and coherence. Our approach further leverages a fine-tuned transformer agent for robust feature extraction and employs an integrated system to align textual and visual content effectively. Experimental evaluations on the CNN/DailyMail and IndiaToday datasets demonstrate that RLMS outperforms state-of-the-art multimodal summarization approaches, achieving higher cosine similarity, ROUGE scores, and better image–summary alignment. By framing summarization within a reinforcement learning paradigm, RLMS provides a scalable and generalizable solution for generating concise, coherent, and contextually enriched multimodal summaries.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

RLMS: A Reinforcement Learning-based Multimodal Summarization Approach for Multimodal Input Data

  • 期刊论文

  • 发表日期：

    2026年5月31日

查看全文
  • Siginamsetty Phani,
  • Ashu Abdul

摘要

Multimodal summarization methods often depend on supervised learning, which limits their generalization to unseen data. To address this, we propose Reinforcement Learning-based Multimodal Summarization (RLMS), a novel framework that generates multimodal output (MO) summaries text paired with contextually relevant images from multimodal input (MI) data containing both text and images. RLMS is structured into three layers: (1) a Multimodal Feature Extraction (MFE) layer that captures latent features from text and images, (2) a Reinforcement Learning (RL) layer that generates textual summaries using policy gradients, and (3) a Multimodal Summarization (MS) layer that integrates the generated text with the most relevant image to form the final summary. The RL layer introduces a summarization-based policy gradient (SPG) as a reward function to directly optimize summary quality, ensuring contextual accuracy and coherence. Our approach further leverages a fine-tuned transformer agent for robust feature extraction and employs an integrated system to align textual and visual content effectively. Experimental evaluations on the CNN/DailyMail and IndiaToday datasets demonstrate that RLMS outperforms state-of-the-art multimodal summarization approaches, achieving higher cosine similarity, ROUGE scores, and better image–summary alignment. By framing summarization within a reinforcement learning paradigm, RLMS provides a scalable and generalizable solution for generating concise, coherent, and contextually enriched multimodal summaries.