Structural Health Monitoring (SHM) is pivotal in identifying and mitigating damage risks in large structures, thereby ensuring their safety and longevity. In the realm of structural vibration-based damage identification, Computer vision (CV)-aided SHM systems offer significant advantages over conventional sensor systems, including the capability to monitor extensive and inaccessible areas with lower installation costs. The present work introduces a novel approach for non-contact CV-based structural vibration response measurement, damage localization and severity quantification from vibration video data. The videos are pre-processed using homography transformation in situations where the image and the structural planes are skewed with respect to each other. The Features from Accelerated Segment Test (FAST) algorithm, with an adaptive threshold strategy, is employed to identify high-quality features within the Region of Interest (ROI) surrounding each sensor location. Subsequently, these feature points are tracked across each video frame using the Kanade-Lucas-Tomasi (KLT) algorithm, to obtain the vibration response in physical units. A hybrid combination of unsupervised learning model, Principal Component Analysis (PCA) and a stochastic subspace state-space system identification algorithm, N4SID is used to extract the state-space matrices from the displacement response. Further Stochastic damage locating vector (SDLV) approach, aided by an improved Bayesian inference-based stochastic model updating (BI-SMU) algorithm, is employed to precisely identify the damaged elements of the structure. The proposed approach is validated on a numerical truss model with simulated damage scenarios and then implemented on the damage localization for a photo-realistic synthetic truss bridge model from its vibration videos. The method shows excellent accuracy in effectively locating structural damages across varying severity levels, thereby enhancing its potential in practical SHM applications.