Accurate prediction of protein secondary structure is a fundamental challenge in bioinformatics and is critical for understanding protein function. In this study, we propose a novel architecture for protein secondary structure prediction that fuses contextual representations from ESM-2 and ProtBERT model using an attention mechanism combined with gating layer. The gating mechanism adaptively controls the contribution of each model’s features, enabling more informative fusion before passing the combined representation through a bidirectional LSTM and a final classification layer. Our custom FusionAttention model achieves a validation accuracy of 88.1%, macro precision of 87.6%, recall of 88.1%, F1-score of 87.8%, and \(\text {R}^2\) of 0.55 on our dataset. We conducted extensive comparisons with individual pretrained models (ESM-2, ProtBERT, ProtT5, and RITA) as well as their average ensemble and weighted ensemble and it consistently outperformed both individual and ensemble approaches, highlighting the effectiveness of incorporating attention and gating mechanisms for multi-model fusion in protein structure prediction tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Protein Secondary Structure Prediction Using Attention-Based Fusion of Language Models

  • Umme Israt Afroz,
  • Neelima Monjusha Preeti,
  • Nazmul Siddique

摘要

Accurate prediction of protein secondary structure is a fundamental challenge in bioinformatics and is critical for understanding protein function. In this study, we propose a novel architecture for protein secondary structure prediction that fuses contextual representations from ESM-2 and ProtBERT model using an attention mechanism combined with gating layer. The gating mechanism adaptively controls the contribution of each model’s features, enabling more informative fusion before passing the combined representation through a bidirectional LSTM and a final classification layer. Our custom FusionAttention model achieves a validation accuracy of 88.1%, macro precision of 87.6%, recall of 88.1%, F1-score of 87.8%, and \(\text {R}^2\) of 0.55 on our dataset. We conducted extensive comparisons with individual pretrained models (ESM-2, ProtBERT, ProtT5, and RITA) as well as their average ensemble and weighted ensemble and it consistently outperformed both individual and ensemble approaches, highlighting the effectiveness of incorporating attention and gating mechanisms for multi-model fusion in protein structure prediction tasks.