Medical image segmentation, especially for tasks such as tumor or lesion delineation, remains challenging due to the inherent subjectivity and ambiguity, resulting in substantial variability across annotations from different human raters. While certain inter-rater differences can offer valuable complementary information that enhances the model’s robustness, large discrepancies may introduce conflicting training signals, ultimately hindering model learning and generalization. To address this, this paper proposes a novel Mixture-of-experts U-Net (MoE U-Net) designed specifically for multi-rater medical image segmentation. Our core idea is to explicitly model individual annotation styles through a weighted fusion of shared expert-generated meta-segments, to handle significant inter-rater variability. Specifically, our framework employs a residual encoder to extract multi-scale image representations, which are then decoded into meta-segments by a set of expert decoders. Gating networks equipped with channel-wise attention dynamically weight these meta-segments to mimic the unique annotation styles of different experts. This design enables diverse annotations to be represented through soft, adaptive combinations of shared meta-segments. We conducted experiments on two public multi-rater datasets and the results demonstrate that our proposed MoE U-Net model achieves superior performance, establishing a new benchmark for multi-rater medical image segmentation.