Parkinson’s disease (PD) is a neurodegenerative disorder that manifest progressive motor impairments. Smooth pursuit eye movement (SPEM) analysis has emerged as a potential PD biomarker, even at prodromal stages. Nonetheless, the standard protocols for SPEM analysis limit the study and discovery of abnormal eye movement patterns. Particularly, the protocols are invasive, and only recover global eye motion trajectories, losing kinematic information. This work introduces a video markerless representation that include a geometric attention mechanism with the capability to learn SPEM parkinsonism patterns under scenarios with limited training samples. This strategy first involves 3D convolutional layers to compute a volumetric bank of activations. Subsequently, the spatial and temporal relationships are synthesized into a symmetric positive definite (SPD) matrix. Then, a geometrical attention mechanism is introduced to identify the most significant feature relationships, even at specific moments in time, preserving the Riemannian geometry. The proposed approach was validated on a study with 15 patients with PD and 15 controls, achieving an average Recall and Precision scores of 0.80, and 0.89 respectively.