Knowledge-based visual question answering (VQA) aims to enhance VQA systems with relevant external knowledge to answer visually grounded questions. The key to solving this task lies in retrieving the correct knowledge from the database. Current retrieval-augmented VQA (RA-VQA) studies primarily focus on retrieving purely textual knowledge, overlooking information from the co-occurring images in knowledge. To address this limitation, we propose extending such knowledge retrieval to multimodal documents. Moreover, another challenge faced by RA-VQA systems is that the retrieved knowledge is not always reliable or effective, while the knowledge contained in training data is assumed to be correct. To address such knowledge inconsistency, we introduce a balanced training strategy to enhance the robustness. Finally, We present RA-VQA+, a novel system based on powerful multimodal large language models, comprising two key components: (1) A knowledge retriever that effectively captures relevance across multimodal query and documents. (2) An answer generator that robustly processes the top-k retrieved knowledge. We conduct extensive experiments on the widely used benchmarks, i.e., E-VQA and InfoSeek. Our multimodal retriever significantly improves knowledge retrieval performance by a large margin. Our final VQA model achieves new state-of-the-art results.