It is still a big challenge to identify tiny surface cracks in a rugged weld due to the lift-off variations using the nondestructive testing (NDT) method. In this paper, the signal gradient algorithm is presented to identify the tiny surface crack in the rugged weld using the alternating current field measurement (ACFM) technique. The ACFM simulation model and testing system was set up to obtain the insensitive signal to the lift-off variations. The signal gradient algorithm was presented to process the insensitive signal for the identification of the tiny surface crack in the rugged weld. The results show that the Bz signal is the insensitive signal to lift-off variations caused by the rugged weld. The signal to noise ratio (SNR) of the crack identification signal was greatly improved by the signal gradient algorithm, and a tiny surface crack can be identified effectively in the weld and the heat affected zone (HAZ).