This chapter presents a comprehensive overview of Recurrent Neural NetworksRecurrent neural network (RNN) (RNNs) and Long Short-Term Memory (LSTM)Long short-term memory network (LSTM) networks, and their extensionsExtension, which were the dominant sequence modeling approaches before TransformersTransformer. Today, they are still used in certain applications (like speech and streaming), and the recurrent principle has re-emerged in modern State Space ModelsStatestate space model (SSM) (SSMs) such as S4Structured state space (S4) and MambaMamba, which compete with TransformersTransformer for long-sequence efficiency. RNNsRecurrent neural network (RNN) are dynamic models designed to handle sequential data by incorporating recurrence, where outputs from previous time steps influence future computations. Despite their usefulness, RNNsRecurrent neural network (RNN) struggle with long-term dependencies due to the vanishing and exploding gradient problems, which motivated the development of more robust architecturesArchitecture. The chapter begins with the fundamentals of RNNsRecurrent neural network (RNN) and the Backpropagation Through TimeBackpropagationbackpropagation through time (BPTT) (BPTT) training algorithm. It then examines key challenges in training RNNs and surveys several mitigation strategies, including leaky units, close-to-identity weight matrices, and echo state networksEcho state network. To effectively model both short-term and long-term dependencies, LSTMLong short-term memory network (LSTM) networks were introduced, followed by Gated Recurrent UnitsGated recurrent units (GRU) (GRUs), which simplify LSTMLong short-term memory network (LSTM) while maintaining its performance. In contexts where sequences can be processed offline, bidirectionalBidirectional architecturesArchitecture outperform their unidirectional counterparts by leveraging both future and past contexts. This leads to the introduction of bidirectionalBidirectional RNNsRecurrent neural network (RNN) and LSTMsLong short-term memory network (LSTM), culminating in models like ELMo that demonstrate the power of bidirectionalBidirectional contextual representationsRepresentation. Sections of the chapter detail the core concepts of RNNsRecurrent neural network (RNN), challenges in learning long-term dependencies, architectural innovations such as LSTMLong short-term memory network (LSTM) and GRUGated recurrent units (GRU), and finally bidirectionalBidirectional models. This chapter serves as a foundational reference for understanding the evolution and capabilities of recurrent architecturesArchitecture in deepLearningdeep learning learningDeepdeep learning.

Recurrent Neural Networks and Long Short-Term Memory Networks

章节
发表日期：

2026年

查看全文

Benyamin Ghojogh,
Ali Ghodsi

摘要

This chapter presents a comprehensive overview of Recurrent Neural NetworksRecurrent neural network (RNN) (RNNs) and Long Short-Term Memory (LSTM)Long short-term memory network (LSTM) networks, and their extensionsExtension, which were the dominant sequence modeling approaches before TransformersTransformer. Today, they are still used in certain applications (like speech and streaming), and the recurrent principle has re-emerged in modern State Space ModelsStatestate space model (SSM) (SSMs) such as S4Structured state space (S4) and MambaMamba, which compete with TransformersTransformer for long-sequence efficiency. RNNsRecurrent neural network (RNN) are dynamic models designed to handle sequential data by incorporating recurrence, where outputs from previous time steps influence future computations. Despite their usefulness, RNNsRecurrent neural network (RNN) struggle with long-term dependencies due to the vanishing and exploding gradient problems, which motivated the development of more robust architecturesArchitecture. The chapter begins with the fundamentals of RNNsRecurrent neural network (RNN) and the Backpropagation Through TimeBackpropagationbackpropagation through time (BPTT) (BPTT) training algorithm. It then examines key challenges in training RNNs and surveys several mitigation strategies, including leaky units, close-to-identity weight matrices, and echo state networksEcho state network. To effectively model both short-term and long-term dependencies, LSTMLong short-term memory network (LSTM) networks were introduced, followed by Gated Recurrent UnitsGated recurrent units (GRU) (GRUs), which simplify LSTMLong short-term memory network (LSTM) while maintaining its performance. In contexts where sequences can be processed offline, bidirectionalBidirectional architecturesArchitecture outperform their unidirectional counterparts by leveraging both future and past contexts. This leads to the introduction of bidirectionalBidirectional RNNsRecurrent neural network (RNN) and LSTMsLong short-term memory network (LSTM), culminating in models like ELMo that demonstrate the power of bidirectionalBidirectional contextual representationsRepresentation. Sections of the chapter detail the core concepts of RNNsRecurrent neural network (RNN), challenges in learning long-term dependencies, architectural innovations such as LSTMLong short-term memory network (LSTM) and GRUGated recurrent units (GRU), and finally bidirectionalBidirectional models. This chapter serves as a foundational reference for understanding the evolution and capabilities of recurrent architecturesArchitecture in deepLearningdeep learning learningDeepdeep learning.