Even though transformer-based networks have achieved remarkable progress in Image Manipulation Localization (IML), they may overlook subtle artifacts and forensic traces which are crucial for precisely identifying the manipulated regions in images. To address this issue, a Locality-guided Transformer Network (LoTraNet) for IML is proposed in this paper. In LoTraNet, a hierarchical dual-level distillation is developed to provide the transformer with multi-stage guidance while in each stage point-level and patch-level distillation are included, and a CNN is used as teacher model to enhance the local feature representation capabilities of transformers. Meanwhile, an attention guidance module is designed in patch-level distillation to explicitly provide adaptive locality guidance for the transformer within patches, which effectively bridges the gap between transformer and CNN. Our analysis and experimental results on various datasets have demonstrated that our proposed network can pay more attention to the local context while maintaining global modeling capabilities and have enhanced ability to locate manipulated regions at any scale compared to state-of-the-art image manipulation localization methods.