Scholarlink

We introduce MME-VirtualWorld, a programmable benchmark for Multimodal Large Language Model (MLLM) Evaluation, designed to address several common issues: 1) the overlap between MLLM training data and benchmark data sources; 2) the lack of precise control over image content; and 3) noise or high costs associated with annotations. In this work, we propose a novel data generation approach through an interactive environment for computer vision research, enabling controllable and realistic image creation. Using this environment, we generated 100K images with a resolution of 2560 \(\,\times \,\) 1440 along with their labels. From these, 13K images were selected to create 32K question-answer pairs, covering 12 vision-related tasks, such as spatial relationship reasoning. We evaluated the performance of 21 models across all tasks, revealing significant limitations in current MLLMs and highlighting potential directions for future optimization. Additionally, we explored task-specific fine-tuning to improve spatial reasoning, achieving a maximum accuracy improvement of 63.25%.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MME-VirtualWorld: Simplifying Multimodal Assessment via Programmable Synthetic Benchmarking

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Zedong Liu,
  • Li Chen,
  • Shenao Chen,
  • Xiang He,
  • Aoqi Fu,
  • Jiaxiang Liu,
  • Shikun Feng

摘要

We introduce MME-VirtualWorld, a programmable benchmark for Multimodal Large Language Model (MLLM) Evaluation, designed to address several common issues: 1) the overlap between MLLM training data and benchmark data sources; 2) the lack of precise control over image content; and 3) noise or high costs associated with annotations. In this work, we propose a novel data generation approach through an interactive environment for computer vision research, enabling controllable and realistic image creation. Using this environment, we generated 100K images with a resolution of 2560 \(\,\times \,\) 1440 along with their labels. From these, 13K images were selected to create 32K question-answer pairs, covering 12 vision-related tasks, such as spatial relationship reasoning. We evaluated the performance of 21 models across all tasks, revealing significant limitations in current MLLMs and highlighting potential directions for future optimization. Additionally, we explored task-specific fine-tuning to improve spatial reasoning, achieving a maximum accuracy improvement of 63.25%.