A significant challenge in visual understanding-based assistive robotics has been targeted action recognition in human–robot collaborative (HRC) spaces. With vision-based data, there exist certain irregularities and variabilities in the working environment that can considerably influence recognition performance. In most cases, high-dimensional vision sensor data is first projected to a low-dimensional manifold and then analyzed for supervised or unsupervised recognition. Classical dimensionality reduction (DR) techniques are mostly affected by such variations in the data, whereas the family of local structure-sensitive manifold learning techniques such as the 2D locality preserving projections (2DLPP) offers a better alternative. However, the projection map generated by 2DLPP also heavily relies on the spatial structure and intensity variation in the visual data. To overcome these concerns, a cosine similarity measure is introduced for calculating similarity information between data points in the originally structured image space, with enhanced efficiency. Additionally, a granular computing-based local, intrinsic structure encoding mechanism is incorporated that works on granular information fusion and rough entropy maximization. The granular fusion scheme considers RGB color spaces separately to allow individual channel granular encoding from the images. Extensive experimental studies demonstrate the proposed approach to be an effective visual cue detection method, especially in shape and color-varying scenarios.