The article presents the use of a convolutional neural network with a U-Net architecture for the automatic detection of areas designated for cladding in the repair and refurbishment process of machine parts. The main assumption is that the operator marks the wear area on the cleaned metallic surface using a colored marker. This area is then detected using the robot’s vision system and its cooperating neural network. Accurately identifying the marked area allows for vectorizing the acquired data and adjusting it to the robot’s spatial coordinates. This enables the robot to streamline the cladding process by automating the tool path generation. The article presents a comparison of the effectiveness of the U-Net network with various parameters. Additionally, it demonstrates how the learning process depends on the input data being provided as either RGB or HSV format images. The obtained results confirm that the use of this image segmentation architecture can be very helpful in the process of automatically detecting markings on a cleaned metallic surface.