Attention is All You Need (Vaswani et al., Attention is all you need, 2017) represents a watershed moment in machine learning. Cited over 130,000 times at the time of writing, it introduced the transformer, an architecture built through the repeated application of self-attention layers. Fast, flexible, and remixable, this architecture has taken over machine learning, starting with natural language processing. But what does self-attention in transformers have to do with attention, as studied by psychologists and neuroscientists? In this chapter, we trace the history of this pivotal architecture back to its distant neuroscience and artificial intelligence inspirations, focusing on three lines of work: attention to align sequences, attention as differentiable computing, and visual attention over images. We show how the insights distilled from these lines of work, embodied in the transformer architecture, are now helping to answer interesting neuroscientific questions: discovering what drives our innate visual biases for shape versus texture, modeling eye movements and salience, and understanding how memories are stored and related in the hippocampus. We conclude with ideas about how to build the future of neuroscience and artificial intelligence.