The early diagnosis of breast cancer is of crucial importance for patient treatment. However, existing breast tumor datasets suffer from limited labeled sample sizes and general object detection algorithms fail to explicitly model boundary information for specific scenarios such as breast tumor detection. Therefore, this paper presents an innovative unsupervised learning approach, and proposes two key components: 1) an Explicit Boundary Coarse Modeling (EBCM) module and 2) a Boundary-Aware Layer (BALayer) with frequency-domain separation, which collaboratively enhance the extraction of discriminative breast tumor features. The core idea lies in leveraging the powerful capability of unsupervised learning to automatically extract the tumors’ salient features, while enhancing semantic extraction through BALayer, thereby providing rich discriminative features for downstream medical tasks. Experimental results demonstrate that our method achieves significant improvement by 15.2% on mAP in unsupervised breast tumor detection and 9.0% mAP in semi-supervised scenarios. It can more accurately identify tumor regions and significantly enhance the precision and recall of breast tumor detection. Moreover, it can also serve as a pre-trained model with discriminative features for breast tumor identification on small-sample breast tumor datasets.