Protein Secondary Structure Prediction Using Attention-Based Fusion of Language Models
摘要
Accurate prediction of protein secondary structure is a fundamental challenge in bioinformatics and is critical for understanding protein function. In this study, we propose a novel architecture for protein secondary structure prediction that fuses contextual representations from ESM-2 and ProtBERT model using an attention mechanism combined with gating layer. The gating mechanism adaptively controls the contribution of each model’s features, enabling more informative fusion before passing the combined representation through a bidirectional LSTM and a final classification layer. Our custom FusionAttention model achieves a validation accuracy of 88.1%, macro precision of 87.6%, recall of 88.1%, F1-score of 87.8%, and \(\text {R}^2\) of 0.55 on our dataset. We conducted extensive comparisons with individual pretrained models (ESM-2, ProtBERT, ProtT5, and RITA) as well as their average ensemble and weighted ensemble and it consistently outperformed both individual and ensemble approaches, highlighting the effectiveness of incorporating attention and gating mechanisms for multi-model fusion in protein structure prediction tasks.