Computer-aided diagnosis is poised to image-based detection of molecular alterations, potentially improving the dilemma of time-consuming and costly clinical genetic testing. Prevalent approaches employ a two-stage scheme, which first identifies tumor regions from whole slide images (WSIs) and then exhaustively learns patches split from these areas with a patch-wise supervision model under WSI labels to characterize WSIs. However, parts of the patches are uninformative, and spatially close patches may exhibit similar discriminative power. Moreover, a model must traverse all possible magnifications to select the optimal one, as various cancer subtypes may necessitate different optimal recognition magnification. All these constraints lead to significant inefficiencies in the application of deep models on large histopathology images. To alleviate these limitations, this paper proposes a joint magnification and attention sampling-based cascade network for BRCA mutation detection from histopathology images. Specifically, to tackle redundant patches and tedious traversal, large regions from WSIs are downsampled to generate region-wise attention maps, which are the basis for selecting representative patches from different spatial locations at different magnifications. Meanwhile, to enhance the effectiveness of patch-level representations, the cascade feature fusion network (CFF) is proposed to integrate high-level representations from early stages to facilitate low-level feature learning in subsequent stages. Experimental results on the public ovarian dataset TCGA show our method can effectively reduce resource requirements and run 2.6–6.5 times faster in inference with a slight classification performance loss compared to the SOTA methods.