Language, being the most important tool of communication, has encouraged researchers to solve the issues related to cross-language communication. As a result, neural machine translation (NMT) has become a highly researched matter, and significant development has been made in the domain recently. This paper gives an overview of the LSTM-based and the transformer-based encoder-decoder neural architectures in MT, which are the most predominant ones in recent times. The concept of attention concerning NMT, which allows selective focus on some parts of the sentence to improve NMT, has also been discussed. Though NMT has worked well for high-resource languages, translating low-resource languages remains among the major challenges in machine translation, the reason being the unavailability of an appropriate corpus for such languages. In this context, the transfer learning approach has shown notable results. The evolution of the transfer learning approach concerning machine translation has also been explored.