Colorectal cancer is a malignant tumor or cancer of the large intestine. According to the report produced by the WHO in 2018, colorectal cancer, which affects both sexes equally, is the third most frequent type of cancer. It ranks third among males and second among females in terms of cancer incidence. The course of treatment known as a biopsy is typically done to look at any irregularities on the surface of the colon by removing a piece of surgical sample that has been processed and fixed onto glass slides is used as the source of material for the study of histopathology, which looks at the disease symptoms found through microscopic analysis. The analysis of histopathological images is very important for the identifying the colorectal cancer either as benign or malignant stage. The proposed system using a deep learning technique based on ResNet101 to classify adenocarcinoma (malignant) from benign tissues. While building the convolutional Neural Network (CNN) model on a massive dataset yields decent classification accuracy, but the training process is computationally expensive. As a result, we also look into how transfer learning methodologies might be used in our work. This in the results, transfer learning based ResNet101 model considerably outperforms the CNN model, fully trained on colorectal cancer samples and obtains an accuracy of about 99.06%. Finally, the proposed classification model highlights its superiority and improves classification accuracy compared with state-of-art models.