Lung and colon cancers pose significant healthcare challenges globally due to their high mortality rates. Recent research has extensively focused on using machine learning and computer vision techniques to classify these diseases based on histopathological images. However, simple CNN architectures often struggle to extract effective features from complex backgrounds, leading to suboptimal feature representation and poor Accuracy. Additionally, it is necessary to focus on different parts of the input image depending on the class being considered to improve classification Accuracy. To address these challenges, this study presents a new framework known as class-specific attention-based convolutional neural networks (CSAB-CNN). This architectural design aims to categorize three distinct types of lung tissues and two distinct types of colon tissues. The class-specific attention mechanism plays an effective role in this approach by allowing the model to focus on the most relevant features for each tissue type. This targeted attention boosts the model’s capability to discern subtle differences in histopathological features unique to each class. For the training and evaluation of the proposed model, we used the LC25000 dataset, which contains complex histopathological images classified into five classes. We employed the proposed CSAB-CNN model for thorough classification and analyzed the features using LIME and Grad-CAM visualizations. Our proposed model achieved impressive results: 100% training Accuracy, 99.64% validation Accuracy, Precision, and Recall, a kappa score of 99.55% and an AUC of 99.99%. Class-wise metrics further emphasize our approach’s effectiveness in accurately distinguishing between lung and colon cancer.