Deep Learning Algorithms are becoming predominant in Remote sensing, particularly in Hyperspectral Images (HSI), while Convolutional Neural Networks (CNN) and their variations are widely employed due to their proficiency in extracting local features, their limitations in capturing sequential properties of spectral characteristics have led to increased attention on transformers. In computer vision, there’s a notable shift towards Vision Transformer (ViT)-based models. However, these models lack image-specific inductive bias, such as translational equivariance and locality. The application of transformers in HSI faces a critical challenge due to the absence of suitable pre-processing and optimization methods, significantly impacting the overall performance of the models. To address this, we adopt a ViT-based backbone and incorporate pre-processing strategies inspired by spectral–spatial residual networks (SSRN) to effectively extract and analyze the spectral and spatial features of hyperspectral data. By combining these techniques, the proposed model demonstrates a significant improvement in accuracy, even when trained on fewer samples compared to previous approaches. The implementation code is available at github.com/ShashNagendra/HSI-Image-Classification-using-CNN-and-ViT .