Labeling inconsistencies, either across annotators (inter-observer variability) or within a single annotator (intra-observer variability), undermine the reliability of supervised learning. These discrepancies arise when subjective judgments assign different labels to the same image, introducing noise that degrades model accuracy. Most noisy-label methods treat errors as random or class-dependent; however, annotation errors reflect systematic biases that correlate with image content and individual annotator tendencies. Conventional approaches cannot capture such structured noise; thus, annotator-aware strategies become essential. We presented the Adaptive Model Integration Network (AMIN), which regarded each annotator as an expert and integrated their predictions according to an estimated reliability score. AMIN assigned a score to every annotator for each image and used these scores to weight annotator-specific model outputs. Then, it distilled the weighted predictions into a unified student model, improving robustness to labeling inconsistencies. We evaluated AMIN in two scenarios: (i) a synthetic CIFAR-10 dataset in which we swapped labels for five class pairs at 10%, 30%, 80%, and 100% to simulate structured noise, and (ii) a five-class lung-adenocarcinoma dataset labeled by five pathologists. For CIFAR-10, AMIN surpassed majority voting at every swap rate, gaining 0.27–0.61% points in macro-F1 for 10–80% swaps and smaller gains at 100%. For the pathology dataset, AMIN outperformed existing methods and increased the invasive-mucinous subtype’s F1 from 0.65 to 0.95. These results indicate that the image-wise integration of annotator-trained models of AMIN improves robustness to both inter- and intra-observer variability, providing a solution for tasks plagued by subjective or inconsistent annotations.