Scholarlink

Abstract <p>Vision-language models (VLMs) such as GPT-4, LLaVA, and CogVLM have recently gained widespread adoption due to their impressive performance on several vision and language related tasks. However, current assessment methods miss an important component: uncertainty, which is critical to a comprehensive assessment of vision and language models. To address this shortcoming, we present a benchmark that incorporates quantitative uncertainty assessment in the evaluation of vision and language models. Our analysis covers over 20 models of vision and language, focusing on the visual multiple choice question and answer task. We examine the models on 5 datasets that assess different vision and language capabilities. Using conformal prediction as an approach to uncertainty assessment, we demonstrate that the uncertainty of models is not consistent with their accuracy. In particular, we show that models with the highest accuracy may also have the highest uncertainty, confirming the importance of measuring it for models of vision and language. Our empirical findings also reveal a correlation between model uncertainty and its linguistic part. The code is available at <a href="https://github.com/EnSec-AI/VLM-Uncertainty-Bench">https://github.com/EnSec-AI/VLM-Uncertainty-Bench</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluation of Multimodal Image and Text Processing Models from an Uncertainty Perspective

  • 期刊论文

  • 发表日期：

    2025年9月1日

查看全文
  • V. V. Kostyumov,
  • B. M. Nutfullin,
  • O. G. Pilipenko

摘要

Abstract

Vision-language models (VLMs) such as GPT-4, LLaVA, and CogVLM have recently gained widespread adoption due to their impressive performance on several vision and language related tasks. However, current assessment methods miss an important component: uncertainty, which is critical to a comprehensive assessment of vision and language models. To address this shortcoming, we present a benchmark that incorporates quantitative uncertainty assessment in the evaluation of vision and language models. Our analysis covers over 20 models of vision and language, focusing on the visual multiple choice question and answer task. We examine the models on 5 datasets that assess different vision and language capabilities. Using conformal prediction as an approach to uncertainty assessment, we demonstrate that the uncertainty of models is not consistent with their accuracy. In particular, we show that models with the highest accuracy may also have the highest uncertainty, confirming the importance of measuring it for models of vision and language. Our empirical findings also reveal a correlation between model uncertainty and its linguistic part. The code is available at https://github.com/EnSec-AI/VLM-Uncertainty-Bench.