Hybrid visual encoders have shown promise in enhancing vision-language alignment for multimodal large language models (MLLMs), yet their potential in small-scale MLLMs remains unexplored. This paper presents Gating-TinyLLaVA, the first compact MLLM that synergizes dual visual encoders (e.g., SigLIP and ResNet) through an adaptive gating network. Our key innovation lies in dynamically fusing complementary visual features—global semantic cues and local structural details—while suppressing error propagation via learnable gating weights. Extensive evaluations across five benchmarks (MME, MM-Vet, VQAv2, GQA, MMMU-val) demonstrate that Gating-TinyLLaVA consistently outperforms state-of-the-art small-scale MLLMs in both accuracy and robustness. Ablation studies validate three critical findings: 1) Dual encoders provide stronger visual grounding than single-encoder baselines, 2) The gating mechanism effectively mitigates feature conflicts and model hallucination, and 3) Our design preserves computational efficiency without sacrificing model capacity. Compared to existing methods, the proposed architecture achieves superior trade-offs between performance and resource demands, establishing a new paradigm for deploying multimodal AI on edge devices.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Gating-TinyLLaVA: A Compact Multimodal Model with Dual Visual Encoders

  • Xuanang Zhang,
  • Yuezhong Wu,
  • Lingjiao Chen,
  • Xi Chen

摘要

Hybrid visual encoders have shown promise in enhancing vision-language alignment for multimodal large language models (MLLMs), yet their potential in small-scale MLLMs remains unexplored. This paper presents Gating-TinyLLaVA, the first compact MLLM that synergizes dual visual encoders (e.g., SigLIP and ResNet) through an adaptive gating network. Our key innovation lies in dynamically fusing complementary visual features—global semantic cues and local structural details—while suppressing error propagation via learnable gating weights. Extensive evaluations across five benchmarks (MME, MM-Vet, VQAv2, GQA, MMMU-val) demonstrate that Gating-TinyLLaVA consistently outperforms state-of-the-art small-scale MLLMs in both accuracy and robustness. Ablation studies validate three critical findings: 1) Dual encoders provide stronger visual grounding than single-encoder baselines, 2) The gating mechanism effectively mitigates feature conflicts and model hallucination, and 3) Our design preserves computational efficiency without sacrificing model capacity. Compared to existing methods, the proposed architecture achieves superior trade-offs between performance and resource demands, establishing a new paradigm for deploying multimodal AI on edge devices.