CNO-Former: Chaotic Neural Oscillatory Transformer for Social Media Text Generation
摘要
Transformers have become foundational in NLP applications, including social media dialogue learning and text generation. Large Language Models (LLMs), pre-trained on extensive data, progressively acquire knowledge for diverse NLP tasks. However, vanilla self-attention requires quadratic time and space, posing challenges in balancing computational efficiency and information representation. Models like Linformer, Performer, Reformer, Longformer, and Synthesizer introduce sparse attention and low-rank projections to address this but suffer from significant information loss and low-resolution representations. This paper proposes the Chaotic Neural Oscillatory Transformer (CNO-former), which uses chaotic dynamics in dot-product attention within multi-head and masked multi-head self-attention. This improves computational efficiency, convergence speed, and reduces information loss. Comprehensive testing against traditional and state-of-the-art models demonstrates that CNO-former achieves superior performance in preserving information and minimizing loss. It outperforms benchmarks on BLEU, F1, and PPL metrics for sentence evaluation, underscoring its effectiveness in text generation tasks.