Deep networks are becoming incredibly common analysis in medical imaging such as cancer diagnosis, Radiotherapy, and identifying abnormalities. Skin cancer is mostly consistently category of cancer discovered in light skin color of human beings on the report of one survey. Ultraviolet (UV) radiation is attributed as the leading cause of skin cancer. An Adversary can exploit the deep network to his advantage and can manipulate the output of the network. A fully automated system for skin cancer classification that can offer defense against adversarial attacks is introduced in this study. Adversarial image generation with PGD (Projected Gradient Decent) attacks, FGSM (Fast Gradient Sign Method), transfer learning of a pre-trained Google-net, and adversarial training of the Google-net on adversarial and augmented images are the five important steps. For Transfer learning, the final layers of a pre-trained GoogleNet are replaced for better performance. The network was trained on an image dataset of 1000 classes. The network is trained to classify skin cancer melanoma and benign samples. The proposed methodology provides a defense against adversarial attacks and improves testing accuracy as well.