Accurate chest X-ray diagnosis typically requires a nuanced understanding of image semantics. Hence, masked image modeling (MIM) has gained attention as a promising approach for learning local image semantics by reconstructing masked patches from their surrounding context. However, existing MIM methods have two key limitations when applied to chest X-ray images: (1) the use of semantic-unaware masking strategies which may corrupt clinically significant visual features, and (2) insufficient multi-scale supervision which fails to consider abnormalities of varying sizes. To address these challenges, we introduce CXR-MIM, a novel MIM framework that incorporates clinical visual priors to guide the image modeling process. Specifically, we leverage these priors to indicate key aspects of a visual feature, including its semantics, location, size, and shape. For semantic and location awareness, CXR-MIM utilizes radiologists’ gaze data to differentiate between clinically significant and insignificant regions in the masking phase, implementing a controlled masking strategy that moderately masks significant diagnostic features. To capture abnormalities of different sizes and shapes, we develop a pyramid adaptive reconstruction module to provide supervision across multiple scales in the reconstruction phase, which is further enhanced by semantic-aware recalibrated gaze heatmaps. Experiments on both two publicly available datasets and one private dataset demonstrate the superior performance of CXR-MIM compared to existing MIM methods. Further evaluation involving pre-training on an additional large-scale dataset indicates promising scalability with increasing data size, underscoring its potential in the age of foundation models. Our code will be available at Github .