<p>Anomaly detection from medical images is badly needed for automated diagnosis. For example, medical images obtained with several modalities, such as magnetic resonance (MR) and confocal microscopy, need to be classified for anomaly detection. The MR images may contain tumors that need to be detected with anomaly detection techniques. In MR imaging, brain tumors are represented as areas composed of abnormal cells that need to be detected. On the other hand, confocal microscopy can be used for retinal imaging. Hence, there is a need to apply classification techniques for diagnosing diseases of the eye, such as Diabetic Retinopathy (DR), which leads to exudates in different regions of the retina. The detection of exudates is necessary for the diagnosis of DR. In this paper, two models are presented for the classification of MR and retinal images. Much research has been done in this area, but it did not achieve high accuracy of classification in an acceptable time. In addition, the previous models did not consider both types of image modalities considered in this paper. The suggested models are Convolutional Neural Network (CNN) and Convolutional Long Short-Term Memory (Conv-LSTM). They are used to classify normal and abnormal cases of both brain tumor and DR diseases. The CNN model is composed of six convolutional layers and six max-pooling layers. A soft-max activation function takes the classification decision. The proposed Conv-LSTM model is composed of three stages: pre-processing, feature extraction, and classification. The deep learning model performance is better than those of traditional networks. The CNN gives an accuracy of 94.26% for DR classification and 95.8% for brain tumor classification, while the Conv-LSTM gives the maximum accuracy of 96.41% for DR classification and 97.4% for brain tumor classification.</p>