Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across various multimodal tasks, promising a bright future for artificial intelligence. However, their widespread use is hindered by the persistent issue of object hallucination, where models generate responses misaligned with visual input. To address this, we propose Visual Continuity Disruption (VDIS), a training-free, plug-and-play contrastive decoding method. This approach is inspired by our observation that disrupting visual continuity destabilizes visual tokens, forcing the model to rely more heavily on language priors and statistical biases—key contributors to object hallucinations—during inference. VDIS works by disrupting visual continuity to induce biased distributions based on language priors and statistical biases, then contrasts these disturbed distributions with the original ones to eliminate object hallucination-inducing factors. Comprehensive experiments across several hallucination benchmarks demonstrate the effectiveness of our method in multiple MLLMs.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

VDIS: Combating Object Hallucination in Multimodal Large Language Models

  • Fuchuan Tang,
  • Gaocai Wang

摘要

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across various multimodal tasks, promising a bright future for artificial intelligence. However, their widespread use is hindered by the persistent issue of object hallucination, where models generate responses misaligned with visual input. To address this, we propose Visual Continuity Disruption (VDIS), a training-free, plug-and-play contrastive decoding method. This approach is inspired by our observation that disrupting visual continuity destabilizes visual tokens, forcing the model to rely more heavily on language priors and statistical biases—key contributors to object hallucinations—during inference. VDIS works by disrupting visual continuity to induce biased distributions based on language priors and statistical biases, then contrasts these disturbed distributions with the original ones to eliminate object hallucination-inducing factors. Comprehensive experiments across several hallucination benchmarks demonstrate the effectiveness of our method in multiple MLLMs.