<p>Accurate and precise identification of cholelithiasis is essential for saving the lives of millions of people worldwide. Although several computer-aided cholelithiasis diagnosis approaches have been introduced in the literature, their use is limited because Convolutional Neural Network (CNN) models are black box in nature. Therefore, a novel approach for cholelithiasis classification using custom CNN with post-hoc model explanation is proposed. This paper presents multiple contributions. First, a custom CNN architecture is proposed to classify and predict cholelithiasis from ultrasound image. Second, a modified deep convolutional generative adversarial network is proposed to produce synthetic ultrasound images for better model generalization. Third, a hybrid visual explanation method is proposed by combining gradient-weighted class activation with local interpretable model agnostic explanation to generate a visual explanation using a heatmap. Fourth, an exhaustive performance analysis of the proposed approach on ultrasound images collected from three different Indian hospitals is presented to showcase its efficacy for computer-aided cholelithiasis diagnosis. Fifth, a team of radiologists evaluates and validates the prediction and respective visual explanations made using the proposed approach. The results reveal that the proposed cholelithiasis classification approach beats the performance of state-of-the-art pre-trained CNN and Vision Transformer models. The heatmap generated through the proposed hybrid explanation method offers detailed visual explanations to enhance transparency and trustworthiness in the medical domain.</p>