In the previous chapter, we introduced recurrent models (RNN, LSTM, etc.) as general neural net architectures to model sequential data such as texts. However, RNNs are not powerful enough to learn long-term dependency and complex sequence-to-sequence mapping. In this chapter, we will introduce a new powerful technique for sequence modeling named the attention mechanism and a new family of network architecture, transformers, that builds on this mechanism. The contents will be organized in the following order: first, we will systematically analyze the main issues with vanilla recurrent models and how the attention mechanism can help alleviate them. Then we will introduce self-attention and transformers, a new powerful sequential modeling architecture that has become the most popular choice of models in the NLP domain. Last, we will introduce several pretrained transformer models and how they can be applied to solve different types of NLP and vision tasks.