Sign language is a type of nonverbal communication method used by people with hearing impairments and the deaf to convey their sentiments and emotions. According to the 2011 Census of India data, there are over 50 lakh people in India who are deaf, and nearly 20 lakh people have speech impairments. In this paper, we have presented a technique to recognise the hand movements of Indian Sign Language efficiently by using a hybrid model of a strong convolutional neural network (CNN) and support vector machine (SVM). In this work, an Indian Sign Language (ISL) dataset is used to train the model. This dataset contains various two handed Indian Sign Language images. In the proposed hybrid model, CNN is used to serve as an automated feature extractor that extracts the most informative features, whereas SVM serves as a classifier. The experimental findings show the correctness of the suggested framework by reaching an identification accuracy of 99.78% for this model on the ISL dataset, and after including Gaussian and random noise to the images, the CNN-SVM model’s accuracy drops to 92.72%.