Skin lesion classification presents significant challenges due to complexities such as high inter-class similarity, high intra-class variation, small regions of interest, and data imbalance. Despite the availability of numerous computer-aided skin cancer classification systems based on convolutional neural networks (CNNs), their performance often falls short of dermatologists’ expectations. Fusion of features from various networks and incorporating residual blocks, attention mechanisms, and transfer learning have been used for improved performance of CNNs. In our approach, we address these challenges by proposing a multimodal method that leverages pre-trained DenseNet201, EfficientNet-B0, and Xception as backbone architectures by modifying their input layers and adding new layers to improve their performance through retraining them on skin lesion datasets. A multimodal network fusing their feature maps has been proposed. We explore a layer and a classifier for which the fusion provides the best results without compromising the time complexity. It is observed that the fusion of features obtained from the output of the last convolutional layer, along with the Naïve Bayes classifiers, provides the best performance. The performance of the proposed network is assessed on the ISIC 2018 and 2019 datasets, which demonstrates superiority over the state-of-the-art techniques.