Lung and colon cancer are considered as two of the leading causes of fatalities in human beings. The ability to detect this cancer is very necessary to determine the subsequent actions. In this paper, we introduce to perform lung and colon cancer detection using a framework of multiple state-of-the-art deep learning architectures. The feature maps of three models: Xception, VGG-16, and VGG-19 are combined which will result in a bigger feature map. Then, we train this hybrid model from scratch on a large dataset of Lung and Colon images called LC25000 which consists of five classes with 5000 images per class. Finally, softmax layer of the combined model is used to classify images. Classification results show that our hybrid model achieves a high classification accuracy of 99.34% using four-fold cross-validation. Comparison has also been carried out on previous work experiments performed on the same dataset and results showed that our hybrid model outperforms state-of-the-art methods. Thus, our model has the potential to become a valuable tool in clinics, helping doctors in cancer diagnosis.