We proposed a novel unsupervised graph-based method for estimating the true labels of cervical cytology images, which were annotated by eleven annotators into one of six categories corresponding to their pathological progression stages in accordance with the Bethesda System. Our approach involves constructing a graph based on the similarity between images and then leveraging an unsupervised learning method. Our key idea here is the propagation of categorical distributions from the annotator’s opinion in a label propagation manner across the similarity graph. The experimental dataset is highly noisy, resulting in only approximately 36% of the true labels being assigned. Our proposed method achieved superior estimation accuracy from these highly noisy labels, outperforming both the classical Dawid-Skene algorithm and a recently proposed sample-wise label fusion algorithm. Furthermore, we demonstrated that the estimation accuracy can be significantly enhanced by incorporating the confusion matrix, provided that knowledge of the common misclassifications between labels is available. Python code for our proposed method is publicly available from https://github.com/iida0yasuhiro/Experiment