Towards a Dataset for Estimation of Keyboard Fingerings
摘要
In this paper, we describe a system that combines keyboard detection, key segmentation, piano transcription, and hand-tracking into a pipeline for general piano performance videos recorded from an overhead perspective. Keyboard localization was performed by a fine-tuned YOLOv8 model on manually annotated frame samples, followed by a procedure for segmenting and labeling the keys. Keyboard fingerings were predicted using a distance metric between the hands and key boundaries, achieving an 82% accuracy. Utilizing the knowledge of hand positions demonstrated up to a 6.6% improvement in piano transcription F1 score.