Endoscopic imaging plays a crucial role in modern diagnostics and minimally invasive procedures. However, artifacts caused by specular and diffuse reflections present significant challenges, particularly in tasks such as endoscopic image segmentation. Existing methods tackling endoscopic artifacts typically address only one type of reflection, failing to fully account for the non-Lambertian reflectance of endoscopic tissue structures. Therefore, inspired by the simplified Phong model for endoscopy, we propose a two-stage artifact inpainting framework. The first stage suppresses specular artifacts, while the second stage focuses on inpainting diffuse artifacts. Additionally, we introduce a weight map to control the handling of diffuse artifacts, ensuring a more precise enhancement. To evaluate its effectiveness, we focus on its impact on endoscopic image segmentation tasks. Extensive experiments on multiple colonoscopy and dental endoscopy datasets demonstrate that our framework can robustly improve the segmentation performance of endoscopic images, offering better enhancement than existing state-of-the-art methods. Particularly, for zero-shot SAM segmentation of teeth, a significant performance boost is observed after inpainting, with mDice and mIoU increasing from 51.5%/39.3% to 96.1%/93.0%. Code is available at GitHub .