Deep Learning has become a highly popular technology in the last decade due to its ability to process massive amounts of data with state-of-the-art computational resources. Deep Learning has tremendous potential to be used in medical applications. Architectures such as Convolutional neural networks (CNN) and Recurrent neural networks (RNN) have achieved significant success in tasks such as image classification, text classification, and sequence-to-sequence classification (Jurtz et al. 2017). In this chapter, we discuss selected deep learning architectures that are commonly used for sequence and image data.