Brain age estimation based on structural MRI can serve as a powerful tool for exploring the impact of abnormal neurodegeneration on brain structure. Existing methods primarily focus on biomarkers associated with global structural changes in the brain, making it difficult to localize abnormalities to specific brain tissues. Furthermore, these approaches generally allow models to autonomously learn the relationship between brain structure and brain age from the images, without incorporating effective medical prior knowledge. To address these limitations, we propose TissueAgeNet: a dual-path image-text architecture that integrates imaging data with medical prior information for brain-tissue-level age estimation. Specifically, the proposed method utilizes a segmentation network to extract brain tissue masks, which, along with MRI images, are input into a visual encoder. Concurrently, tissue-level prior attributes—such as morphological and signal intensity features—are transformed into textual representations by a Large Language Model and serve as inputs to the text-stream. Finally, the two modalities are fused via simple linear integration to achieve accurate brain-tissue-level age estimation. We validate our approach on three datasets, including those of fetuses, preterm infants and Alzheimer’s patients. The results demonstrate accurate age prediction across diverse populations, and on the OASIS-3 dataset (Alzheimer’s patients), we show that our model can identify structural neurodegeneration at the tissue level.