Machine learning (ML) has shown strong performance in medical imaging analysis. However, most models rely on large, high-quality annotated datasets, which are time-consuming, costly, and require domain expertise. Recent studies have explored ML-based methods for generating annotations. While promising, these approaches demand considerable computational resources and rely on the initial model’s performance, posing challenges in low- and middle-income countries with limited infrastructure. To address these limitations, we developed a novel, open-source-based tool for efficient and transparent medical image annotation using a human-in-the-loop (i.e., three bioscience students and five neuroradiologists) approach. Although the annotations generated were suitable to train downstream ML models, it is important to evaluate if and how the annotators and clinicians can introduce biases into the datasets when using our tool. Thus, in this work, we analyzed interobserver agreement and used mixed-effect models to identify potential biases in the annotation workflow. Our findings showed that interobserver agreement for intracranial volume segmentations ranged from substantial to nearly perfect, indicating strong consistency among neuroradiologists. Regression analysis revealed that neither annotator identity nor annotation time significantly affected acceptance rates, suggesting that structured training and feedback may have reduced variability. These aspects are essential to build trust and incentivize the community to utilize our tool, which represents a practical and equitable solution for medical imaging research in low-resource settings.