Image restoration has achieved remarkable progress with the advancement of deep learning models. Although Transformer architectures have made significant strides in recent years, window-based self-attention mechanisms still suffer from limited receptive fields and noisy interactions from irrelevant regions. To address these issues, we propose LAT, a Luminance-Aware Transformer for image restoration. LAT introduces a luminance-guided neighborhood attention mechanism that adaptively adjusts the importance of features within each receptive field window based on luminance priors. This allows the attention mechanism to focus more effectively on salient regions, thereby enhancing the extraction of important features. In addition, we incorporate a semantic-guided attention module, which clusters semantic information by computing the similarity between learnable slots and tokens. By masking tokens belonging to different semantic groups, this module reduces interference from irrelevant content within attention windows. Extensive experiments on various image restoration tasks show that LAT adopts a lightweight design while achieving performance comparable to state-of-the-art methods, demonstrating an excellent trade-off between accuracy and computational efficiency.