SBIR allows intuitive image search using sketches, but is difficult, as the modality between sketches and photos has a modality gap and fine-grained instance-level differences are required. We introduce BFS-Net, a bi-level feature selection system, which consists of: (i) Frequency-Enhanced Extraction Module (FEEM) that uses frequency-domain attention to boost high-frequency features, like edges and contours, and (ii) Feature-Similarity Estimation Module (FSEM) that ranks and keeps the most informative sketch-photo tokens. A combination of these modules reconciles the global with the local, eliminating irrelevant noise. BFS-Net, in contrast to previous methods, is loss-agnostic and yields identical performance on triplet/quadruplet, contrastive and circle losses. QMUL-Shoe-V2, QMUL-Chair-V2, and Sketchy benchmark experiments indicate that BFS-Net performs better than state-of-the-art approaches, achieved with 1–2% improvements in Acc@1. The complementary nature of FEEM and FSEM is supported by the results of ablation and BFS-Net can be used as a scaled and effective solution to fine-grained SBIR. The suggested sketch retrieval framework is realistic in the real-world sketch retrieval scenario and can be scaled down to mobile/low power.