Breast cancer is the most prevalent type of cancer worldwide which makes early detection and diagnosis essential. The application of deep learning in medical diagnostics is a real breakthrough, offering powerful methods to help fight cancer. It’s changing how we detect cancer early, making it easier to catch the disease sooner and potentially save more lives. In this research, we provide a distinctive model called ViT-RES, which applies an innovative method known as Attention Exploitation. This approach integrates Vision Transformers and Convolutional Neural Networks to capture both global and local features respectively. The representations from Vision Transformer are parametrically upscaled using a combination of convolutional transpose and standard convolutional layers. After being processed by the upsampling block, the representations are passed to the convolutional layer with minimal parameters. We propose two variants: ViT-RES Base, which merges ViT Base with ResNet-18 and ViT-RES Large, which combines ViT Large with ResNet-18. In binary classification, ViT-RES Base and ViT-RES Large achieved 96% and 98% accuracy respectively on the BreakHis dataset at certain magnifications.