Recently, the number of Alzheimer’s disease patients has increased, and the disease seriously affects their daily lives. Hence, more and more researchers have paid attention to this disease, and the diagnostic technology has been improved, particularly with the application of imaging technologies such as \( ^{18}\text {F-FDG}\) PET. Although many methods based on deep learning have made significant progress in this field, early effective diagnosis remains challenging. Existing research primarily relies on imaging data for disease prediction, but the information that imaging data can provide is limited. Hence, this paper proposes a text-guided method for Alzheimer’s disease prediction. To be specific, vision Mamba is employed as image encoder and a CLIP encoder that can extract long text features is introduced as another branch. Then, an attention module is designed to make the text encoder provide additional semantic information for the image encoder, thus guiding attention distribution and enhancing cross-modal consistency. Finally, the prediction result can be obtained by the features after interaction. Experimental results demonstrate that the proposed method outperforms current state-of-the-art algorithms on the public ADNI dataset.