Deep Neural networks are prevalent in medical image analysis, yet their performance deteriorates when there is a mismatch between the magnification levels of training and testing images. This study evaluates the robustness of various deep learning architectures for breast cancer histopathological image classification under differing magnification scales. We compare CNN-based models like ResNet and MobileNet, self-attention-based Vision Transformers and Swin Transformers, and token-mixing models such as FNet, ConvMixer, MLP-Mixer, and WaveMix. Using the BreakHis dataset, which includes images at multiple magnification levels, we demonstrate that WaveMix achieves stable and high classification accuracy, regardless of magnification differences between training and testing data. Our findings underscore the importance of selecting robust deep learning architectures capable of handling domain shifts, such as magnification variation, to ensure reliable performance in histopathological image analysis. Additionally, we assess the classification performance using popular off-the-shelf, pre-trained computer vision backbones to identify suitable models for medical applications.