Concept Bottleneck Models (CBMs) offer an interpretable framework for deep learning by introducing human-understandable concepts as intermediaries between input data and class predictions. However, the unconstrained learning of concept-to-class mappings often results in unfaithful explanations and shortcut reliance. In this work, we present KL-CBM, a novel CBM architecture that integrates a transparent, probabilistic module with a flexible dense classifier. The transparent head estimates class probabilities based on the alignment between predicted concepts and empirical class-specific profiles. Simultaneously, a dense classifier following the concept layer is softly aligned to the transparent head using Kullback–Leibler (KL) divergence. This encourages predictive consistency with interpretable modeling without sacrificing model capacity. We evaluate KL-CBM on two real-world concept-annotated datasets—AwA2 and aPY—and demonstrate that it achieves strong classification performance and competitive concept prediction accuracy. Our results highlight that KL-CBM maintains semantic alignment between concepts and classes, exhibits robustness to concept-level perturbations, and yields faithful concept-based explanations. Overall, KL-CBM advances the trade-off between interpretability and accuracy, offering a principled and flexible alternative to conventional CBMs.