Negative Sample Optimization and Bias Mitigation for Robust Visual Question Answering
摘要
Currently, VQA faces the challenge of linguistic bias, wherein models frequently depend on surface-level patterns from the training data, rather than generating accurate responses through a thorough analysis of image content. This dependence on dataset bias undermines the robustness and the effectiveness of VQA systems when deployed in practical settings. To alleviate the above problems, we propose network models that use negative samples to optimize and weaken biased features, i.e., NSBM. We construct negative samples using the Negative Sample Synthesis method (NSS), which aids model training by constructing negative samples from randomly sampled questions and images from a small batch of data, forcing the model to concentrate on key entities and relevant words, thus improving visual interpretability and sensitivity to the question. Bias model (BIAM) collaborates with the target model during training to accurately identify biases within the model using adversarial targeting and the loss function. Subsequently, BIAM is integrated with the target model to perform the debiasing process. Our extensive experiments on NSBM on the datasets VQA CPv2, VQA-v2 and VQA CPv1 show that our model is valid.