Manual annotation of bilingual handwritten mathematical answer sheets remains a formidable challenge due to the coexistence of multilingual text, different types of components, diverse handwriting styles, and overlapping symbols. To address this, we propose a unified semi-automatic annotation pipeline combining a tailored customised LabelMe interface with the YOLOv10s to handle this. This unified pipeline automates initial predictions and employs human-in-the-loop refinement to correct errors, significantly reducing manual effort while ensuring scalability without compromising annotation quality. The primary contribution is the introduction of the JUDVLP-MATHANSWERSHEET.v2 dataset, collection of 718 bilingual handwritten mathematical answer sheets, annotated with nine distinct categories, including mathematical expressions, bilingual text, trigonometric diagrams, table, mathematical symbols, operators and numeric etc. Experimental evaluations demonstrate YOLOv10s’ superiority, achieving 73.20% precision and 36.81% mAP@50-95, outperforming other state-of-the-art models. The proposed semi-automatic annotation method reduces annotation time by 74% compared to manual processes, enabling efficient large-scale dataset creation. This study adds to the field of document analysis by providing a unified framework for the detection of handwritten mathematical answer sheets. The work fills a major gap in the automation of the educational sector and the development of multilingual AI applications.