Fourier vs. Attention: A Re-look at Protein Sequence Generation Models
摘要
Protein sequence generation has traditionally relied on autoregressive and diffusion based models, while Fourier-based approaches remain largely unexplored in this domain. We apply FNet, a Fourier transform based architecture originally developed for natural language processing, to protein sequence generation and compare its performance against ESM2’s direct masked sampling strategy and the autoregressive model ProGen. All three models were trained and evaluated on a curated kinase dataset under identical experimental conditions. FNet achieves substantial computational gains over ProGen, reducing generation time by 95% and memory consumption by 39%, while producing sequences with improved amino acid compositional accuracy and competitive structural plausibility scores. ESM2 generates sequences twice as rapidly as FNet, whereas ProGen shows stronger short-range motif preservation. These results indicate that Fourier-based token mixing provides a complementary alternative to attention based and autoregressive generation paradigms for protein sequence modeling, highlighting clear trade-offs between computational efficiency, global sequence statistics, and local motif fidelity. While attention-based models remain superior in overall performance, Fourier-based encoders demonstrate promise as efficient components within future hybrid or large scale protein modeling frameworks, with current limitations largely attributable to FFT implementation overhead rather than architectural constraints. Code available here: https://github.com/bashirgit/Fourier_vs._Attention .