Advancing Smart Healthcare with an Enhanced Multi-Modal for Med-VQA
摘要
Smart healthcare systems are the catalyst for the transformation of urban health services, using advanced technologies to address the growing medical demand in populated areas. This paper presents a new multimodal Medical Visual Question Answering (Med-VQA) framework, designed to improve diagnostic accuracy and efficiency in e-health applications. The proposed approach integrates state-of-the-art models, including CLIP for textual feature extraction, combined with EfficientNet for visual feature extraction. A transformer-based architecture is implemented as a feature fusion to predict answers to medical questions accurately. The evaluation of the proposed approach on two medical datasets, VQA-RAD and PathVQA, demonstrates that the method achieves state-of-the-art performance, with significant improvements in both open-ended and close-ended question modalities. Specifically, the model achieved a superior accuracy of 81.25% on VQA-RAD and 80.39% on PathVQA across diverse question types. These results highlight the potential of the proposed system as a decision-support tool in clinical environments. Finally, we address challenges such as the limitations of medical data and the need for enhanced semantic understanding, flooring the way for future advancements in Med-VQA systems.