Over the past ten years, breast cancer (BC) has become one of the leading causes of cancer-related deaths among women globally. The likelihood of survival can be improved with early identification of breast cancer. Many methods and approaches for early BC detection are now available. Within the breast, anomalies such as micro-calcifications, masses, and asymmetry can be detected and categorized using machine learning (ML) techniques. It is now much simpler to diagnose BC and differentiate tumors from damaged breasts using ML methods. Using an optimal deep learning technique within an IoT health environment framework, this article seeks to categorize breast tumors as benign or malignant. In addition to preventing overfitting and lowering computing costs, the proposed model accurately reduces model complexity. The suggested cascaded architecture includes improving contrast with fast local Laplacian filtering (FlLpF) and transforming whole slide images (WSIs) to tiled images. Then, modified-Xception (m-Xception) replaces the initial convolutions with an architecture and performs the categorization. The model goes a step further by including linear residual-linked depth-separable convolution layers inside the convolution layer. A successful model was trained using a two-stage transfer learning technique. Improved Sand-piper Optimization Algorithm (ISOA) is a metaheuristic that finds near-optimal solutions for large-scale situations by optimally selecting the parameters of the proposed model. Both the HER2SC and HER2GAN datasets validate that the projected perfect outstrips the state-of-the-art alternatives.