A multimodal fusion and symmetric dual-encoder based generation method for KB-VQA
摘要
Knowledge-based visual question answering (KB-VQA) requires the effective integration of visual and textual information, along with external knowledge, to generate answers. During the answer generation phase, traditional methods employ separate encoders to handle information. However, encoding purely in text format results in the loss of visual features, while multimodal encoding cannot fully utilize the rich knowledge present in the natural language space. Other methods attempt to use encoders of different modalities simultaneously, but since the encoders produce outputs in different semantic spaces, it becomes challenging to effectively combine features from different modalities. In this paper, we propose an answer generation method based on multimodal fusion and a symmetric double encoding structure. The method achieves independent encoding and dynamic interaction between image-text modalities by parallelly processing dual-channel visual and textual modal information. Leveraging the advantages of a symmetric architecture, the model employs a cross-modal multi-head attention mechanism to fuse the obtained symmetric multimodal features, thereby generating a global feature vector. This vector serves as a more comprehensive semantic representation that not only closely correlates diverse information types but also provides deeper and more thorough semantic understanding. Experimental results demonstrate that our method significantly outperforms the existing baselines on the OK-VQA dataset, achieving an accuracy of 61.9% and exceeding the state-of-the-art asymmetric methods by (+3.2%), and it also achieves an accuracy rate of 74.85% on the FVQA dataset. Additionally, we conducted comprehensive ablation experiments on each component of the method, systematically proving the advantages of the symmetric multimodal architecture.