<p>Accurate tracking of patients’ hands in wearable-camera videos is important for developing computer-aided tools for rehabilitation assessment. Although recent deep learning models can detect and segment hands in egocentric videos, reliable identification of the patient’s left and right hands remains challenging. In practical rehabilitation scenes, the hands may be blurred, occluded by objects, change shape rapidly, or be confused with the hands of therapists or assistants. This study aims to identify and track the left and right hands of a patient during rehabilitation exercises. The proposed framework first uses Mask R-CNN to segment hand-related regions, including the left hand, right hand, left full-hand, and right full-hand. It then introduces MergeTrack, a graph-based tracking refinement algorithm that improves SORT-based tracking results by linking fragmented hand trajectories across consecutive frames. Finally, a confidence score is computed at each frame by combining the segmentation and tracking results, allowing the system to select the patient’s hands and remove non-patient hands. The proposed method was evaluated on twelve rehabilitation videos collected from patients performing therapy exercises. The results show that the proposed framework improves patient-hand tracking compared with conventional tracking methods, especially in videos with motion blur and occlusion. On average, the number of identity switches is reduced by 50%, and the tracked duration of hand trajectories is approximately doubled. In addition, the detection rate of non-patient hands decreases from 28.1% using Mask R-CNN alone to 2.1% with the proposed framework. These results indicate that the proposed method can provide more stable patient-hand trajectories and support higher-level rehabilitation analysis, such as measuring hand activities and counting hand-object interactions.</p>