Vef-BART: an effective method to mitigate hallucinations through vision enhancement and fusion in BART-based multimodal abstractive summarization
摘要
Multimodal abstractive summarization is a technique for creating short summaries from multiple forms of information. Research based on generative pre-trained language models has made significant progress in summary generation. Most previous methods have limitations in processing visual information and often cannot directly help guide the precise content of the summary. In the process of cross-modal feature fusion, it is often difficult for text to fully capture and understand the overall context of visual information, affecting the authenticity of the generated summary. To overcome these limitations, we design a novel visual encoder that aims to obtain summary-oriented visual features to help generate higher-quality summaries. Our sparse attention mechanism helps focus the source document information on more critical areas in the image to better capture the relevance and importance between modalities. In addition, we introduce a minimum margin loss to suppress the overconfidence problem of the model when generating text during reasoning. We conduct extensive experiments on the public datasets How2 and MMSS. Compared with the baseline model, our method improves the ROUGE-n, ContentF1 and BLEU-n indicators by an average of 5.1%, 2.0% and 4.9% respectively. In further summary quality evaluation experiments, this method improved factual consistency while ensuring summary fluency, and the non-phantom reading rate was increased by 1.7% compared with the baseline model, proving the effectiveness and superiority of this method.