Handwritten Character Recognition (HWCR) requires more features to work with handwritten characters because of their variable and non-uniform nature. Challenges in HWCR increased further when the number of classes increased and mathematical symbols were included. The main aim of this research work is to adapt the pre-trained models to work with multiclass classification for single channel grayscale images of mathematical symbols along with handwritten alphanumeric characters and compare their performances. A customized CNN layer was introduced to convert the single-channel grayscale images to three channels as a preliminary step in recognizing handwritten characters. After introducing the customized CNN layer, various pre-trained models were used as hidden layers to compare the performance of different pre-trained models in recognizing characters in handwritten grayscale images of small size (32 × 32). The separate datasets containing English capital letters, digits and mathematical symbols were merged as a single dataset of 42 classes for this study. EfficientNetB0, ResNet50, MobileNetV2, VGG16, DenseNet121 and conventional CNN models were employed to recognize the character. VGG16, DenseNet121, and CNN were very fast in recognizing the character in the grayscale images. These three models reached about 90% accuracy at the first epoch itself. The test accuracy of these three models was around 96% and higher than the others. Conventional CNN took less time than other models. Conventional VGG16 was better than other models in terms of accuracy.