Scholarlink

Fine-tuning large-scale Vision–Language Models (VLMs) is computationally demanding, motivating the need for efficient data utilization. Existing subset selection methods, such as COINCIDE, primarily focus on distribution matching but overlook instance-level utility, redundancy, and task-specific reasoning relevance. We propose QUBO-based Informative Subset Selection (QUBISS), a unified framework that formulates data selection as a Quadratic Unconstrained Binary Optimization (QUBO) problem. QUBISS jointly maximizes task-relevant data utility and minimizes sample redundancy to promote diversity and compactness. Central to our method is the task vector, which quantifies the semantic contribution of textual information to reasoning performance and integrates it into the QUBO utility term. When applied to fine-tuning LLaVA v1.5, QUBISS selects only 20% of a 665K image–text dataset while achieving results superior or comparable to COINCIDE on both cognition-oriented (MME-C) and perception-oriented (MME-P) benchmarks. The observed gains underscore the value of task-aware semantic guidance for cost-efficient multimodal fine-tuning. Furthermore, advances in large-scale quantum solvers could further enhance QUBISS by directly solving large QUBOs without decomposing them into cluster-level subproblems, thereby mitigating suboptimality arising from problem partitioning.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

QUBO-Based Subset Selection for Efficient Fine-Tuning of Vision–Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Akihiro Yoshida,
  • Keiichiro Yamamura,
  • Hiroki Ishikura,
  • Shinjiro Hirai,
  • Ken Kawano,
  • Yoshihiko Fujisawa,
  • Katsuki Fujisawa

摘要

Fine-tuning large-scale Vision–Language Models (VLMs) is computationally demanding, motivating the need for efficient data utilization. Existing subset selection methods, such as COINCIDE, primarily focus on distribution matching but overlook instance-level utility, redundancy, and task-specific reasoning relevance. We propose QUBO-based Informative Subset Selection (QUBISS), a unified framework that formulates data selection as a Quadratic Unconstrained Binary Optimization (QUBO) problem. QUBISS jointly maximizes task-relevant data utility and minimizes sample redundancy to promote diversity and compactness. Central to our method is the task vector, which quantifies the semantic contribution of textual information to reasoning performance and integrates it into the QUBO utility term. When applied to fine-tuning LLaVA v1.5, QUBISS selects only 20% of a 665K image–text dataset while achieving results superior or comparable to COINCIDE on both cognition-oriented (MME-C) and perception-oriented (MME-P) benchmarks. The observed gains underscore the value of task-aware semantic guidance for cost-efficient multimodal fine-tuning. Furthermore, advances in large-scale quantum solvers could further enhance QUBISS by directly solving large QUBOs without decomposing them into cluster-level subproblems, thereby mitigating suboptimality arising from problem partitioning.