Recent studies have shown notable progress in panoramic image inpainting using deep learning-based methods, particularly Transformer-based approaches. However, conventional Transformer models mostly establish global relationships between features, which may overlook crucial local details such as texture, edges, and structures, essential for realistic image restoration in inpainting tasks. To address this limitation, we propose a novel Transformer architecture called the Hierarchical Transformer (Hi-former). Unlike conventional Transformers, the Hi-former is designed to help the model capture local and global relationships between features, enhancing the model’s overall structural consistency. In addition to the Hi-former, we introduce a Comprehensive Attention Module (CAM) to adaptively learn the significance of different channels and regions during the inpainting process. By dynamically allocating attention to relevant features, CAM enhances the model’s comprehension of the image, leading to improved accuracy and fidelity in inpainting results. Experimental results on the SUN360 dataset demonstrate the effectiveness of our approach, with an average increase of 3.04 dB in PSNR, a 0.0533 improvement in SSIM, and an average decrease of 1.52 in FID score compared to the SOTA method.