To address the limitations associated with camera performance and the smoothness of object surfaces in depth estimation using depth cameras, which can result in issues such as holes or outlier noise in the acquired depth images, existing methodologies for depth estimation frequently incorporate image segmentation techniques. However, many of these studies rely on traditional image processing methods for segmentation, which often exhibit limited applicability and low accuracy. In light of recent advancements in large visual models, this paper introduces a novel deep image restoration approach that employs the Segment Anything Model (SAM). SAM is capable of achieving high-precision instance segmentation across diverse scenes while demonstrating remarkable stability. By leveraging the segmentation results produced by SAM, we propose an enhanced depth image restoration algorithm based on the Fast Marching Method (FMM), which integrates adaptive median filtering to refine depth edge information. Automotive depth maps, characterized by their wide range of lighting conditions, complex scene content, and significant surface reflections, pose significant challenges to depth image restoration techniques. The method has been validated using the Middlebury dataset and real-world automotive imagery, demonstrating its ability to preserve edge details while effectively filling in holes in the depth images. Even when faced with the specific complexities of automotive depth maps, the method has shown exceptional performance and robustness.