Background: Cybercriminals typically employ malware to achieve their objectives, which include botnets, ransomware, etc. Encryption, packaging, and polymorphism make it difficult to identify malware files, especially when they are generated in large quantities every day. The new technique of image-based detection focuses on detecting malware by transforming it into an image and classifying the image as benign or malicious. The raw binary of malware is converted into a grayscale image, and neural networks are used to classify it as benign or malicious, or to classify the image to its respective malware family. Proposed Method: This system proposal introduces a new Penguin Search Optimisation with Deep Learning-based Image Classification (PSODL-IC) for Malware Spectrogram. The presented PSODL-IC technique uses spectrogram images in conjunction with the DL model to classify malware files and differentiate them from benign files. In the presented PSODL-IC technique, the noise elimination stage employs Gaussian filtration (GF). In addition, the presented PSODL-IC method employs the Xception feature extractor in conjunction with a PSO-based hyperparameter optimizer. In this investigation, the auto encoder (AE) is utilised for classification purposes. Outcome and Discussion: Extensive experimental analysis has demonstrated that the PSODL-IC technique yields superior results to other deep learning models, with 97.21 percent accuracy. The obtained results demonstrate the advantages of the PSODL-IC method over other models.