Significant memory concern (SMC), a preclinical phase of Alzheimer’s disease (AD), is at increased risk of underlying AD pathology. Monitoring the progression of SMC is crucial for timely intervention of AD and related disorders. Learning-based neuroimage analysis provides a non-invasive and objective solution for SMC prognosis. However, existing studies usually focus on utilizing imaging data, without considering subjects’ demographic information which is essential for individual-level analysis. In addition, due to the characteristics of SMC progression requiring longitudinal analysis (e.g., 2 years), the data used for SMC analysis are usually very limited (e.g., tens), which poses a huge challenge to model training. To address these limitations, we propose a prompt-driven multi-view learning (PML) framework for predicting the clinical progression of SMC by integrating T1-weighted MRI with demographic information. Specifically, PML comprises four key components: (1) data-driven MRI feature extraction using a residual neural network to learn representative features from 3D MRI scans; (2) handcrafted MRI feature extraction to incorporate domain knowledge on brain tissues (e.g., cortical thickness); (3) demographic feature encoding using a prompt-based strategy through a contrastive language-image pretraining encoder; and (4) feature fusion and classification to jointly model multimodal information. To alleviate data scarcity challenges, we initialize our model with pretrained weights and employ transfer learning to enhance performance. Experimental results on a total of 469 subjects demonstrate the efficacy of PML in predicting SMC progression.