Background <p>Proteins regulate diverse biological processes through interactions with other molecules, including RNAs. RNA-binding proteins (RBPs) are essential regulators of gene expression, forming specific mRNA-protein interactions (mRPIs) that influence mRNA processing, translation, and stability. Recently, deep-learning models have been proposed to predict mRPIs using only sequence information, with some reporting near-perfect accuracy. However, such performance appears inconsistent with the biological complexity of RNA recognition by proteins, which is often influenced by protein tertiary structures that are computationally challenging to predict. In related fields such as protein-protein interaction prediction, data leakage, particularly caused by overlapping proteins between training and test sets, has been shown to substantially inflate performance metrics. Nevertheless, whether similar issues affect mRPI prediction has not yet been systematically investigated.</p> Results <p>We constructed an mRPI benchmark dataset from CLIP experiments and implemented two data partitioning schemes: a random interaction-level split and an RBP-aware split in which pairs of all test RBPs were excluded from training. Three RBP sequence encoding strategies were evaluated within an attention-based deep-learning framework under both partitioning settings: sequence-based one-hot encoding, language model-derived encoding, and structure-aware encoding. Across all models, performance remained high only when test RBPs were also present in the training data. When predicting interactions for unseen RBPs, performance dropped substantially, indicating limited generalization. Even replacing RBPs with their most similar counterparts from the training set did not meaningfully improve generalization. These results suggest that additional protein features beyond sequence information are required to achieve robust mRPI prediction. Overall, our study demonstrated that existing mRPI prediction models are largely overfitted to their original training RBPs and fail to generalize to unseen proteins.</p> Conclusions <p>Overall, we provided a curated benchmark dataset, a rigorous evaluation framework, and an attention-based model that achieves the best generalization performance among currently available methods, with an approximately 8.5% auROC improvement over existing tools. These resources will facilitate the development of more reliable and broadly applicable mRPI prediction tools.</p> Scientific contribution <p>This work presented the first systematic investigation of data leakage and generalization in mRNA-protein interaction prediction, demonstrating that most reported near-perfect performance is largely driven by RBP overlap between training and test sets. By introducing an RBP-aware evaluation framework and a benchmark dataset, we revealed that most sequence-based models fail to generalize to unseen RBPs, even when enhanced with protein language model-derived and structure-aware encodings. Our study established a more rigorous evaluation standard for mRNA-protein interaction prediction, highlighting the critical need for protein diversity and beyond-sequence features to advance reliable mRNA-protein interaction prediction.</p>