Convolutional Neural Networks (CNNs) have significantly improved the performance of brain lesion segmentation. However, accurate segmentation of brain lesions remains challenging when the imaging appearance of lesions is similar to that of normal brain tissue. To address this problem, this study improves brain lesion segmentation by incorporating anatomical priors from healthy subjects’ scans. This prior knowledge enhances the differentiation between lesions and normal brain tissue. To integrate this prior knowledge, we propose registering a set of reference scan images from healthy subjects to each scan image containing lesions. The registered reference scans provide reference intensity samples of normal tissue at each voxel location. In this way, spatially adaptive priors can indicate abnormal voxels, even when their intensity is similar to that of normal tissue, because their anatomical location is inconsistent with the established map of normal tissue. Specifically, through reference scan images, we compute abnormality score maps for scans containing lesions. These abnormality score maps serve as auxiliary inputs to the segmentation network to assist brain lesion segmentation. The proposed strategy was evaluated on different brain lesion segmentation tasks, and the results demonstrate the effectiveness of integrating anatomical prior knowledge using our method.