The classification of skin cancer is quite challenging due to the shortage of well-experienced dermatologists. It has a significant impact on cancer treatment. Early detection of skin cancer allows for effective treatment. Contemporary methods used right now worldwide operate with high efficiency. However, because there are similarities among various types of skin cancer, it is necessary to focus on the specific details of the input images. This poses a significant challenge in accurately categorising them into separate classes. To overcome this issue, the model should pay attention to both global and local features of the images. Thus, this paper uses P2P-Net (Part to Pose Network), an architecture that works besides the backbone and learns about the minor details of the input image using FPN (Feature Pyramid Network). FPN helps the model classify input images into the correct classes. It regularises the weight vectors using the features of the image patches extracted using FPN. This approach utilises a graph-based representation of objects to uncover the global configuration of the local parts. The study applied P2P-Net to the skin cancer dataset with ResNet50 as the backbone and achieved an average accuracy of 94.76%.