As multi-modal data becomes more prevalent in disease diagnosis, leveraging it effectively is essential for enhancing diagnostic outcomes. Among existing methods that utilize multimodal data, existing methods often ignore the negative impact that some modalities may have on the overall model. Moreover, the effective extraction of intra-modal features presents a significant challenge for the model. In pursuit of this objective, we introduce a versatile Multi-Modal Multi-View Graph Convolutional Network (MMMV-GCN) for diagnosing Alzheimer’s Disease. To mitigate the adverse impact of inter-modality interactions, we employ a multi-view model that uses voting systems for classification, reducing the potential for negative influence between modalities. To capture intra-modal details, we introduce a transformer-based feature extractor that efficiently retrieves information from individual views. Furthermore, we incorporate demographic data to enhance the model’s performance and interpretability. This approach outperformed contemporary state-of-the-art techniques in diagnosing both diseases across multiple scenarios.