Predicting mental states from neural data holds potential for various real-world applications, such as brain–computer interfaces. However, the high costs of data acquisition generally limit the size of these data, which hinders the application of advanced deep neural networks (DNNs) to the prediction of mental states. To address this issue, we proposed a self-supervised pretraining framework for DNNs using large-scale neural data. Specifically, we pretrained a DNN using functional magnetic resonance imaging (fMRI) signals from over 800 individuals in the Human Connectome Project (HCP) database. The pretrained DNN was then applied in the ICBHI 2024 Scientific Challenge (ICBHI-SC), in which the cross-participant prediction of emotional classes and ratings is performed using a small dataset—only 30 trials per participant—collected during video watching. After the DNN was fine-tuned, emotion prediction was performed using features extracted by the DNN. Finally, we confirmed that the prediction performance surpassed that of baseline models without the use of self-supervised pretraining. Thus, our framework potentially provides a versatile tool for mental state prediction from limited neural data, and expands the applicability of such predictions in real-world situations.