Transformer models have achieved groundbreaking results across AI fields such as NLP and CV but incur \(\mathcal {O}(n^2)\) cost from self-attention, hampering long-sequence scalability. This survey reviews architectures designed to overcome these constraints, encompassing both optimized Transformer variants and wholly different paradigms. We begin by outlining the need for alternatives, emphasizing the difficulty of processing very long inputs. We then organize recent models into three groups: (i) structured state-space approaches (e.g., S4) exploiting continuous-time formulations to realize linear complexity; (ii) pure MLP schemes (e.g., MLP-Mixer) that forgo attention in favor of learned token-mixing layers; and (iii) convolution–attention hybrids (e.g., CoAtNet) that fuse locality biases with global receptive fields. For each class, we dissect design principles and analyze theoretical scaling. A comparative evaluation highlights trade-offs in accuracy, compute, and domain suitability. We close by discussing open issues—robustness, scaling, hardware co-design—and suggest avenues for advancing efficient sequence modeling. Our findings guide the development of next-generation, resource-aware sequence models.