A medical visual question-answering model based on multi-scale feature fusion and question Feature enhancement
摘要
The core task of Medical Visual Question Answering (Med-VQA) is to generate accurate answers to clinical questions based on medical images. However, existing models often struggle to simultaneously extract fine-grained details and global semantics due to their reliance on single-scale feature processing. In addition, most methods model the interaction between questions and visual features using bilinear attention, failing to fully leverage visual information to enhance text representations, which leads to suboptimal cross-modal fusion. To address these issues, we propose a novel multi-scale and cross-modal enhancement framework for Med-VQA. Our approach introduces two key innovations: (1) A multi-scale visual encoder that combines vision transformer (ViT) and convolutional neural network (CNN) features through a global–local fusion mechanism, enabling the model to simultaneously capture fine-grained details and global semantics, overcoming the limitations of single-scale processing. (2) A question enhancement module that dynamically refines text representations using visual information before multimodal fusion, enabling more effective cross-modal interactions. By jointly optimizing multi-scale visual feature fusion and text enhancement, our approach more comprehensively integrates image details and global information, effectively capturing cross-modal interactions. This enables the model to better understand the complex relationships between images and text, thereby improving reasoning accuracy. Extensive experiments on the VQA-RAD and SLAKE benchmark datasets show that our model achieves an accuracy of 77.2% on the VQA-RAD dataset and 83.1% on the SLAKE dataset.