The flexibility, versatility and enhanced perception of visio-tactile sensors could be beneficial for advanced robotic systems and other applications requiring precise haptic feedback. In this paper, we present a comprehensive framework that combines material classification and haptic feedback through the use of GelSight sensors. The study includes the creation of a diverse material dataset, consisting of 13 material classes of 42 distinct indoor and outdoor items, each item with multiple video samples captured over different regions and pressing conditions by human-held GelSight mini sensors. We introduce a method for detecting pressing events from recorded video samples and extracting key frames that capture important material features. We employ both traditional and deep learning-based feature extraction techniques to model material characteristics. These features are then used to classify materials with high accuracy through supervised learning methods using different image resolutions. For the traditional approach, Histogram of Oriented Gradients (HOG) feature descriptor combined with SVM gives 95.41% accuracy using the new image dataset. After fine-tuning five pre-trained models for transfer learning on our dataset, Dense-Net121 (96.42%), InceptionV3 (96.32%), and VGG16 (95.77%) models show promising results, and the ensemble of these 3 fine-tuned models provides highest accuracy of 97.63%. The results demonstrate the feasibility of using GelSight-based material classification for haptic feedback, with potential implications for virtual reality, robotic manipulation, and human-computer interaction applications.