Automatic fish classification using machine learning is pivotal for advancing marine biology, sustainable fishery management, environmental conservation, and enforcing fishing regulations. This study develops a technique integrating channel and spatial attention mechanisms into the convolutional feature maps of some pre-trained deep learning networks with class-specific weight learning. Traditional image classification models often struggle with imbalanced datasets and the presence of dominant background features, which can obscure relevant details. This study uses an imbalance carp dataset, ‘Fish-Pak’ consisting of images of major and exotic carp. To address the class imbalance challenge, we propose a framework leveraging channel and spatial attention to enhance feature selection and discrimination capabilities. Different state-of-the-art pre-trained deep learning algorithms such as DenseNet121, MobileNetV2, VGG16, VGG19, and ResNet-50 are explored in different batch sizes and learning rates with a transfer learning paradigm. We designed two network architectures named Carp-Trans and Carp-Trans-Attention to classify the carp fish. With the Carp-Trans-Attention network, DenseNet121 with the transfer learning paradigm achieved the highest recognition performance of 96.29% by an individual network with a batch size of 32. Whereas, the same accuracy is obtained by the Carp-Trans network using the VGG16 backbone with a batch size of 8. A voting based ensemble of these two networks achieved a state-of-the-art accuracy of 98.14% in the ‘Fish-Pak’ dataset. Focusing on the model’s generalization ability, convergence speed, and robustness, an ablation study is performed on variations in batch size, loss function, and learning rate. This study paves the way for further application of attention mechanisms in different deep learning algorithms in aquaculture research and other computer vision domains.