Scholarlink

This study investigates integrating auditory and visual data in computational models, focusing on challenges like audio-visual question answering (AVQA). Current methods often separate audio and visual processing, leading to scalability issues and high computational costs. By leveraging transformer-based architectures like Lin et al.’s LAVisH framework, we propose novel techniques to improve AVQA, especially in musical AVQA tasks. Our method combines advanced image decomposition with a contrastive meta-learning framework to enhance LAVisH for better AVQA performance. Empirical results show a 1.54% improvement for visual questions and 1.2% for audio-visual questions on the MUSIC-AVQA dataset. Furthermore, our framework demonstrates commendable performance by testing AVQA tasks on video lectures, given the problem’s complexity and lack of alternative solutions. By substituting RGB overlapping patches with isotropic sparse decomposition, we expand the number of trainable parameters, facilitating wider application across AVQA tasks beyond musical AVQAs with minimal computational overhead. This framework thus extends its utility to a broader spectrum of AVQA tasks while maintaining low computational costs.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Contrastive Meta-learning Approach with Isotropic Sparse Decomposition for Scalable Audio-Visual Learning

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Dhruv Dixit,
  • Paritosh Pandey,
  • Raman Jha,
  • Pranshul Bhatnagar,
  • Shashank Mouli Satapathy

摘要

This study investigates integrating auditory and visual data in computational models, focusing on challenges like audio-visual question answering (AVQA). Current methods often separate audio and visual processing, leading to scalability issues and high computational costs. By leveraging transformer-based architectures like Lin et al.’s LAVisH framework, we propose novel techniques to improve AVQA, especially in musical AVQA tasks. Our method combines advanced image decomposition with a contrastive meta-learning framework to enhance LAVisH for better AVQA performance. Empirical results show a 1.54% improvement for visual questions and 1.2% for audio-visual questions on the MUSIC-AVQA dataset. Furthermore, our framework demonstrates commendable performance by testing AVQA tasks on video lectures, given the problem’s complexity and lack of alternative solutions. By substituting RGB overlapping patches with isotropic sparse decomposition, we expand the number of trainable parameters, facilitating wider application across AVQA tasks beyond musical AVQAs with minimal computational overhead. This framework thus extends its utility to a broader spectrum of AVQA tasks while maintaining low computational costs.