Accurate prediction of stable and progressive mild cognitive impairment is crucial for early intervention in Alzheimer’s disease. Multimodal data, including clinical tabular data and MRI scans, provide complementary information, yet effective integration remains a challenge. Existing fusion methods, such as cross-attention, emphasize shared modality information while often overlooking critical modality-specific information. In this paper, we propose a novel fusion model that incorporates a hybrid attention module and a latent similarity divergence loss to effectively integrate shared and modality-specific features in a balanced manner. Within the hybrid attention module, self-attention is employed for modality-specific feature learning, while bidirectional cross-modal attention is introduced to extract shared features from clinical tabular and MRI image data. To model clinical tabular data effectively, we propose a column embedding block pre-trained on a large NC-MCI-AD dataset. It captures disease-relevant features while also modeling missing data patterns, making it a robust and essential component for downstream tasks. To model anatomical structure and spatial relationships, image-level features are extracted from MRI data using FastSurferCNN and converted into graph-level representations to capture interactions between brain regions, enabling a richer understanding of disease-relevant patterns. Extensive experiments on ADNI dataset demonstrate that our method outperforms state-of-the-art methods, achieving 0.9042 balanced accuracy and 0.9403 AUC for sMCI and pMCI classification.