Hybrid CNN-LSTM with Attention Mechanism for Medical Visual Question Answering
摘要
Medical Visual Question Answering (MedVQA) has become an important research field at the convergence of medical imaging and natural language processing for its potential to assist medical practitioners by providing precise answers to image-related questions. This paper introduces a hybrid framework combining CNN and LSTM with an attention mechanism for question answering over medical images. The framework processes medical images and interprets natural language questions to generate relevant answers. The multimodal architecture uses a Convolutional Neural Network (CNN) for extracting features from the image and a Long-Short-Term Memory (LSTM) network for handling the question and answer sequences. The Multimodal fusion, in the framework was performed using an attention mechanism. The model was trained and evaluated on two datasets: ImageCLEF VQA-MED 2019 and VQA-RAD 2019. On VQA-MED 2019 dataset, the model achieved training accuracy of 98%, validation accuracy of 61% and testing accuracy of 60% across 69 classes. For the VQA-RAD dataset, the model attained a training accuracy of 98%, a validation accuracy of 70%, and a testing accuracy of 74% across 8 classes.