Deep learning models for medical imaging are prone to learning shortcut solutions that rely on spurious correlations instead of clinically meaningful features, leading to poor generalization to new data. We propose an oracle-guided training scheme that encourages a student model to learn robust features in the presence of shortcuts. Our method regulates prediction confidence across intermediate network layers, significantly reducing shortcut impact. We evaluate our approach on CIFAR10, CheXpert, and ISIC 2017 datasets using ResNet18 and DenseNet121 architectures. Consistently, we outperform a model trained using Empirical Risk Minimization on a dataset containing a shortcut. In several cases, we close the gap on our clean baseline to the point that there is no statistically significant difference in performance. We also address the practical challenge of obtaining a clean oracle model, enhancing the method’s real-world applicability.