Deploying deep learning (DL) models in medical applications relies on predictive performance and other critical factors, such as conveying trustworthy predictive uncertainty. Uncertainty estimation (UE) methods provide potential solutions for evaluating prediction reliability and improving the model confidence calibration. This paper introduces Learning from EXpert Disagreement for UE (LEXU) for medical image segmentation, a method that leverages the variability in annotations from multiple experts to guide model training. By focusing on regions of disagreement among experts and incorporating multi-rater optimization strategy, LEXU enhances the model’s awareness of challenging cases, resulting in better calibration and predictive uncertainty. The method shows a 55% improvement in correlation with expert disagreements at the image level and a 23% improvement at the pixel level, along with competitive segmentation performance compared to state-of-the-art techniques, all while requiring only a single forward pass.