This paper explores the application of deep learning techniques to classify neurological brain disorders, a vital research domain with significant implications for diagnosis and treatment. A robust framework was designed to effectively classify neurological conditions using medical images, including scans of the brain. The approach integrates multi-modal data fusion, incorporating structural, functional, and diffusion-weighted neuroimaging to comprehensively analyse brain anatomy. The model’s ability to distinguish disorders such as Parkinson’s disease and Alzheimer’s disease was revealed in extensive experiments. Insight into the underlying pathological processes can be offered by interpretability methods, like attention mechanisms. Improved diagnostic tools and personalized therapeutic strategies will benefit patient care and outcomes as a result of these advancements.