Medical Large Vision-Language Models (Med-LVLMs) have shown promise in enhancing medical diagnosis by enabling interactive and knowledge-driven healthcare applications. However, these models often suffer from factual hallucinations which may lead to incorrect diagnoses. Retrieval-augmented generation (RAG) has been proposed to mitigate these issues, yet its effectiveness in multi-modal medical applications is hindered by over-reliance on retrieved data and the opacity of text-based reasoning. To address these challenges, we propose GoCa, a multi-modal RAG system based on chain-of-thought (CoT) distillation and explicit thought optimization, which is designed to enhance both the factuality and explainability of Med-LVLMs. Our GoCa consists of three key components: (1) a self-evolving CoT framework that leverages multi-agent collaboration to refine diagnostic reasoning iteratively and (2) a seamless, preference-guided optimization mechanism that distills high-quality CoT reasoning using preference tuning and (3) an adaptive Monte Carlo-like top-k selection strategy. These innovations ensure that the RAG process remains logically transparent and adaptable, significantly improving consistency when integrating retrieve contexts. Experimental results across multiple datasets on medical visual question answering (Med-VQA) demonstrate that GoCa outperforms several recent state-of-the-art methods, achieving superior factual accuracy and coherence. The code can be found at https://github.com/Da1daidaidai/Goca